climateprediction.net home page
Task 11817801

Task 11817801

Name famous_vjwv_1999_200_006711573_3
Workunit 6914826
Created 26 Aug 2010, 17:21:24 UTC
Sent 7 Nov 2010, 5:00:45 UTC
Report deadline 6 Feb 2011, 12:27:56 UTC
Received 9 Nov 2010, 20:34:43 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1087198
Run time 2 days 5 hours 15 min 14 sec
CPU time 2 days 3 hours 24 min 1 sec
Validate state Invalid
Credit 3,119.13
Device peak FLOPS 2.64 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-apple-darwin
Stderr
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Sun Nov  7 15:34:37 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x95e1a1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3ec edx: 0x95db40fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc428 esp: 0xbfffc3ec
   ss: 0x0000001f efl: 0x00000206 eip: 0x95db40fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libirc.dylib
0x909b4000 - 0x909b7fff /usr/lib/system/libmathCommon.A.dylib
0x93920000 - 0x9392efff /usr/lib/libz.1.dylib
0x95db3000 - 0x95f59fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11256, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Mon Nov  8 12:19:24 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x95e1a1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3ec edx: 0x95db40fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc428 esp: 0xbfffc3ec
   ss: 0x0000001f efl: 0x00000206 eip: 0x95db40fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libirc.dylib
0x909b4000 - 0x909b7fff /usr/lib/system/libmathCommon.A.dylib
0x93920000 - 0x9392efff /usr/lib/libz.1.dylib
0x95db3000 - 0x95f59fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19983, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Mon Nov  8 14:56:09 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x95e1a1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3ec edx: 0x95db40fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc428 esp: 0xbfffc3ec
   ss: 0x0000001f efl: 0x00000206 eip: 0x95db40fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libirc.dylib
0x909b4000 - 0x909b7fff /usr/lib/system/libmathCommon.A.dylib
0x93920000 - 0x9392efff /usr/lib/libz.1.dylib
0x95db3000 - 0x95f59fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21082, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Mon Nov  8 14:56:11 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x95e1a1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3ec edx: 0x95db40fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc428 esp: 0xbfffc3ec
   ss: 0x0000001f efl: 0x00000206 eip: 0x95db40fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libirc.dylib
0x909b4000 - 0x909b7fff /usr/lib/system/libmathCommon.A.dylib
0x93920000 - 0x9392efff /usr/lib/libz.1.dylib
0x95db3000 - 0x95f59fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21082, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Mon Nov  8 14:56:13 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x95e1a1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3ec edx: 0x95db40fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc428 esp: 0xbfffc3ec
   ss: 0x0000001f efl: 0x00000206 eip: 0x95db40fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libirc.dylib
0x909b4000 - 0x909b7fff /usr/lib/system/libmathCommon.A.dylib
0x93920000 - 0x9392efff /usr/lib/libz.1.dylib
0x95db3000 - 0x95f59fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21082, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Tue Nov  9 13:59:33 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x95e1a1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3ec edx: 0x95db40fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc428 esp: 0xbfffc3ec
   ss: 0x0000001f efl: 0x00000206 eip: 0x95db40fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vjwv_1999_200_006711573/lib/libirc.dylib
0x909b4000 - 0x909b7fff /usr/lib/system/libmathCommon.A.dylib
0x93920000 - 0x9392efff /usr/lib/libz.1.dylib
0x95db3000 - 0x95f59fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30817, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
 (30911): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Nov 2010 20:11:05 1087198 11817801 famous_vjwv_1999_200_006711573_3 945,386 183,753 0.1944
09 Nov 2010 19:09:24 1087198 11817801 famous_vjwv_1999_200_006711573_3 936,026 181,908 0.1943
09 Nov 2010 18:07:43 1087198 11817801 famous_vjwv_1999_200_006711573_3 926,666 180,074 0.1943
09 Nov 2010 17:36:50 1087198 11817801 famous_vjwv_1999_200_006711573_3 917,306 178,255 0.1943
09 Nov 2010 17:05:50 1087198 11817801 famous_vjwv_1999_200_006711573_3 907,946 176,441 0.1943
09 Nov 2010 16:34:58 1087198 11817801 famous_vjwv_1999_200_006711573_3 898,586 174,627 0.1943
09 Nov 2010 16:04:10 1087198 11817801 famous_vjwv_1999_200_006711573_3 889,226 172,814 0.1943
09 Nov 2010 15:33:19 1087198 11817801 famous_vjwv_1999_200_006711573_3 879,866 170,997 0.1943
09 Nov 2010 15:02:26 1087198 11817801 famous_vjwv_1999_200_006711573_3 870,506 169,183 0.1944
09 Nov 2010 14:31:28 1087198 11817801 famous_vjwv_1999_200_006711573_3 861,146 167,368 0.1944
09 Nov 2010 14:00:22 1087198 11817801 famous_vjwv_1999_200_006711573_3 851,786 165,555 0.1944
09 Nov 2010 13:29:16 1087198 11817801 famous_vjwv_1999_200_006711573_3 842,426 163,743 0.1944
09 Nov 2010 12:57:37 1087198 11817801 famous_vjwv_1999_200_006711573_3 833,066 161,930 0.1944
09 Nov 2010 12:30:27 1087198 11817801 famous_vjwv_1999_200_006711573_3 823,706 160,117 0.1944
09 Nov 2010 11:55:44 1087198 11817801 famous_vjwv_1999_200_006711573_3 814,346 158,303 0.1944
09 Nov 2010 11:24:51 1087198 11817801 famous_vjwv_1999_200_006711573_3 804,986 156,492 0.1944
09 Nov 2010 10:53:51 1087198 11817801 famous_vjwv_1999_200_006711573_3 795,626 154,681 0.1944
09 Nov 2010 10:22:51 1087198 11817801 famous_vjwv_1999_200_006711573_3 786,266 152,868 0.1944
09 Nov 2010 09:51:51 1087198 11817801 famous_vjwv_1999_200_006711573_3 776,906 151,054 0.1944
09 Nov 2010 09:22:58 1087198 11817801 famous_vjwv_1999_200_006711573_3 767,546 149,240 0.1944


©2024 cpdn.org