climateprediction.net home page
Task 12020796

Task 12020796

Name famous_w84t_599_200_006757229_1
Workunit 6960545
Created 13 Nov 2010, 0:57:41 UTC
Sent 13 Nov 2010, 1:07:51 UTC
Report deadline 12 Feb 2011, 8:35:02 UTC
Received 22 Dec 2010, 18:25:48 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 989329
Run time 9 days 7 hours 50 min 48 sec
CPU time 8 days 18 hours 5 min 41 sec
Validate state Invalid
Credit 2,377.97
Device peak FLOPS 1.16 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-pc-linux-gnu
Stderr
<core_client_version>6.10.17</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7685, iMonCtr=1
Model crash detected, will try to restart...
 (7685): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (7802): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
 (1356): No heartbeat from core client for 30 sec - exiting
 (1356): No heartbeat from core client for 30 sec - exiting
 (1356): No heartbeat from core client for 30 sec - exiting
 (1356): No heartbeat from core client for 30 sec - exiting
 (1356): No heartbeat from core client for 30 sec - exiting
 (1356): No heartbeat from core client for 30 sec - exiting
 (1356): No heartbeat from core client for 30 sec - exiting
 (1356): No heartbeat from core client for 30 sec - exiting
 (1356): No heartbeat from core client for 30 sec - exiting
 (1356): No heartbeat from core client for 30 sec - exiting
 (1356): No heartbeat from core client for 30 sec - exiting
 (1356): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Signal 15 received, exiting...
 (1627): called boinc_finish
CPDN Monitor - Quit request from BOINC...
 (1335): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
 (1275): called boinc_finish
SIGSEGV: segmentation violation
Stack trace (7 frames):
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x809e59c]
[0x739400]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x804f906]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x805085a]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x8050ad6]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x2bbbd6]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(__gxx_personality_v0+0xe1)[0x804c449]

Exiting...
 (1221): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
 (1360): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Signal 15 received, exiting...
 (1730): called boinc_finish
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
 (1365): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
 (1309): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Signal 15 received, exiting...
 (1713): called boinc_finish
SIGSEGV: segmentation violation
Stack trace (7 frames):
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x809e59c]
[0x1fb400]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x804f906]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x805085a]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x8050ad6]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x2b3bd6]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(__gxx_personality_v0+0xe1)[0x804c449]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
 (1674): called boinc_finish
 (1259): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Signal 15 received, exiting...
 (1684): called boinc_finish
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
 (1279): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
 (1665): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
 (1732): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
 (1759): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
 (1783): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (1901): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (1940): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (1965): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2004): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2149): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2188): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2212): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2251): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2290): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2544): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2613): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2743): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2767): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2807): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2831): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2855): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
 (1321): called boinc_finish
SIGSEGV: segmentation violation
Stack trace (7 frames):
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x809e59c]
[0xa3c400]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x804f906]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x805085a]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x8050ad6]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x446bd6]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(__gxx_personality_v0+0xe1)[0x804c449]

Exiting...
Signal 15 received, exiting...
 (1520): called boinc_finish
CPDN Monitor - Quit request from BOINC...
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
 (1344): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
 (1208): called boinc_finish
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
 (1212): called boinc_finish
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
 (1209): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
 (1265): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
 (1280): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
 (1407): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
 (2177): called boinc_finish
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
 (1318): called boinc_finish
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
 (1318): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (1318): No heartbeat from core client for 30 sec - exiting
 (1318): No heartbeat from core client for 30 sec - exiting

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
 (1272): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
 (1321): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
OPEN:  File Open Failed: Read-only file system
OPEN:  Unable to Open File dataout/w84tla#pb000000646c1+ for Read/Write

Model crashed: STWORK  : Error opening output PP file on unit 61                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Could not open shared memory...
Error in attaching shared memory region!
Signal 11 received, exiting...
 (1878): called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1699, iMonCtr=1
Model crash detected, will try to restart...
Could not open shared memory...
Error in attaching shared memory region!
Signal 11 received, exiting...
 (1882): called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1699, iMonCtr=1
Model crash detected, will try to restart...
Could not open shared memory...
Error in attaching shared memory region!
Signal 11 received, exiting...
 (1898): called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1699, iMonCtr=1
Model crash detected, will try to restart...
Could not open shared memory...
Error in attaching shared memory region!
Signal 11 received, exiting...
 (1899): called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1699, iMonCtr=1
Model crash detected, will try to restart...
Could not open shared memory...
Error in attaching shared memory region!
Signal 11 received, exiting...
 (1900): called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1699, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
 (1699): called boinc_finish
Signal 15 received, exiting...
 (1333): called boinc_finish
CPDN Monitor - Quit request from BOINC...
 (1216): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
 (1280): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
 (1335): called boinc_finish
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
CPDN Monitor - Quit request from BOINC...
 (1927): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2009): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2141): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2173): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Signal 3 received, exiting...
SIGSEGV: segmentation violation
 (1266): called boinc_finish
Stack trace (7 frames):
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x809e59c]
[0xf30400]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x804c7b4]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x80501eb]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x8050ad6]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x76fbd6]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(__gxx_personality_v0+0xe1)[0x804c449]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
22 Dec 2010 19:41:05 989329 12020796 famous_w84t_599_200_006757229_1 720,746 752,709 1.0443
22 Dec 2010 19:41:05 989329 12020796 famous_w84t_599_200_006757229_1 711,386 742,990 1.0444
17 Dec 2010 12:07:07 989329 12020796 famous_w84t_599_200_006757229_1 702,026 733,195 1.0444
17 Dec 2010 12:07:07 989329 12020796 famous_w84t_599_200_006757229_1 692,666 723,397 1.0444
16 Dec 2010 22:38:13 989329 12020796 famous_w84t_599_200_006757229_1 683,306 713,597 1.0443
16 Dec 2010 19:50:24 989329 12020796 famous_w84t_599_200_006757229_1 673,946 703,798 1.0443
16 Dec 2010 17:05:48 989329 12020796 famous_w84t_599_200_006757229_1 664,586 693,859 1.0440
15 Dec 2010 16:45:52 989329 12020796 famous_w84t_599_200_006757229_1 655,226 684,141 1.0441
15 Dec 2010 14:18:19 989329 12020796 famous_w84t_599_200_006757229_1 645,866 674,401 1.0442
15 Dec 2010 14:18:19 989329 12020796 famous_w84t_599_200_006757229_1 636,506 664,615 1.0442
15 Dec 2010 14:18:19 989329 12020796 famous_w84t_599_200_006757229_1 627,146 654,790 1.0441
14 Dec 2010 21:57:50 989329 12020796 famous_w84t_599_200_006757229_1 617,786 644,824 1.0438
14 Dec 2010 19:29:32 989329 12020796 famous_w84t_599_200_006757229_1 608,426 635,057 1.0438
14 Dec 2010 19:29:32 989329 12020796 famous_w84t_599_200_006757229_1 599,066 625,285 1.0438
14 Dec 2010 19:29:32 989329 12020796 famous_w84t_599_200_006757229_1 589,706 615,513 1.0438
13 Dec 2010 23:49:40 989329 12020796 famous_w84t_599_200_006757229_1 580,346 605,611 1.0435
13 Dec 2010 21:01:45 989329 12020796 famous_w84t_599_200_006757229_1 570,986 595,876 1.0436
13 Dec 2010 18:29:06 989329 12020796 famous_w84t_599_200_006757229_1 561,626 586,147 1.0437
13 Dec 2010 15:35:19 989329 12020796 famous_w84t_599_200_006757229_1 552,266 576,418 1.0437
13 Dec 2010 09:22:22 989329 12020796 famous_w84t_599_200_006757229_1 542,906 566,584 1.0436


©2024 cpdn.org