climateprediction.net home page
Task 11088983

Task 11088983

Name hadsm3dhet2_junn_006602725_4
Workunit 6806098
Created 15 Mar 2010, 12:10:50 UTC
Sent 8 Jun 2010, 4:45:26 UTC
Report deadline 21 May 2011, 10:05:26 UTC
Received 8 Mar 2011, 15:33:47 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1080604
Run time 23 days 7 hours 32 min 5 sec
CPU time 14 days 0 hours 18 min 5 sec
Validate state Invalid
Credit 6,649.31
Device peak FLOPS 2.05 GFLOPS
Application version UK Met Office HadSM3 Slab Model v6.09
i686-apple-darwin
Stderr
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
MainError:	04:27:25 AM	No files match the supplied pattern.
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
MainError:	03:39:00 AM	No files match the supplied pattern.
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting

Model crashed: 

Model crashed: 

Model crashed: 

Model crashed: 

Model crashed: 

Model crashed: 
Sorry, too many model crashes! :-(
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/dataout/restart.day
forrtl: severe (24): end-of-file during read, unit 5, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/jobs/climate.cpdc, line 1, position 0
Image              PC        Routine            Line        Source             
libirc.dylib       00B8B5E3  Unknown               Unknown  Unknown
libirc.dylib       00B8A2D3  Unknown               Unknown  Unknown
libifcoremt.dylib  008C5EFA  Unknown               Unknown  Unknown
libifcoremt.dylib  0083C2E0  Unknown               Unknown  Unknown
libifcoremt.dylib  008678BD  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0024CEFE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0027E075  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0029A2BE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000023B2  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000022D9  Unknown               Unknown  Unknown
Unknown            00000003  Unknown               Unknown  Unknown
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70453, iMonCtr=1
Model crash detected, will try to restart...
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/dataout/restart.day
forrtl: severe (24): end-of-file during read, unit 5, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/jobs/climate.cpdc, line 1, position 0
Image              PC        Routine            Line        Source             
libirc.dylib       00B8B5E3  Unknown               Unknown  Unknown
libirc.dylib       00B8A2D3  Unknown               Unknown  Unknown
libifcoremt.dylib  008C5EFA  Unknown               Unknown  Unknown
libifcoremt.dylib  0083C2E0  Unknown               Unknown  Unknown
libifcoremt.dylib  008678BD  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0024CEFE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0027E075  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0029A2BE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000023B2  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000022D9  Unknown               Unknown  Unknown
Unknown            00000003  Unknown               Unknown  Unknown
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70453, iMonCtr=1
Model crash detected, will try to restart...
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/dataout/restart.day
forrtl: severe (24): end-of-file during read, unit 5, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/jobs/climate.cpdc, line 1, position 0
Image              PC        Routine            Line        Source             
libirc.dylib       00B8B5E3  Unknown               Unknown  Unknown
libirc.dylib       00B8A2D3  Unknown               Unknown  Unknown
libifcoremt.dylib  008C5EFA  Unknown               Unknown  Unknown
libifcoremt.dylib  0083C2E0  Unknown               Unknown  Unknown
libifcoremt.dylib  008678BD  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0024CEFE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0027E075  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0029A2BE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000023B2  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000022D9  Unknown               Unknown  Unknown
Unknown            00000003  Unknown               Unknown  Unknown
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70453, iMonCtr=1
Model crash detected, will try to restart...
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/dataout/restart.day
forrtl: severe (24): end-of-file during read, unit 5, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/jobs/climate.cpdc, line 1, position 0
Image              PC        Routine            Line        Source             
libirc.dylib       00B8B5E3  Unknown               Unknown  Unknown
libirc.dylib       00B8A2D3  Unknown               Unknown  Unknown
libifcoremt.dylib  008C5EFA  Unknown               Unknown  Unknown
libifcoremt.dylib  0083C2E0  Unknown               Unknown  Unknown
libifcoremt.dylib  008678BD  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0024CEFE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0027E075  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0029A2BE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000023B2  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000022D9  Unknown               Unknown  Unknown
Unknown            00000003  Unknown               Unknown  Unknown
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70453, iMonCtr=1
Model crash detected, will try to restart...
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/dataout/restart.day
forrtl: severe (24): end-of-file during read, unit 5, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/jobs/climate.cpdc, line 1, position 0
Image              PC        Routine            Line        Source             
libirc.dylib       00B8B5E3  Unknown               Unknown  Unknown
libirc.dylib       00B8A2D3  Unknown               Unknown  Unknown
libifcoremt.dylib  008C5EFA  Unknown               Unknown  Unknown
libifcoremt.dylib  0083C2E0  Unknown               Unknown  Unknown
libifcoremt.dylib  008678BD  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0024CEFE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0027E075  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0029A2BE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000023B2  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000022D9  Unknown               Unknown  Unknown
Unknown            00000003  Unknown               Unknown  Unknown
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70453, iMonCtr=1
Model crash detected, will try to restart...
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/dataout/restart.day
forrtl: severe (24): end-of-file during read, unit 5, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadsm3dhet2_junn_006602725/jobs/climate.cpdc, line 1, position 0
Image              PC        Routine            Line        Source             
libirc.dylib       00B8B5E3  Unknown               Unknown  Unknown
libirc.dylib       00B8A2D3  Unknown               Unknown  Unknown
libifcoremt.dylib  008C5EFA  Unknown               Unknown  Unknown
libifcoremt.dylib  0083C2E0  Unknown               Unknown  Unknown
libifcoremt.dylib  008678BD  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0024CEFE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0027E075  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  0029A2BE  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000023B2  Unknown               Unknown  Unknown
hadsm3_um_6.09_i6  000022D9  Unknown               Unknown  Unknown
Unknown            00000003  Unknown               Unknown  Unknown
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70453, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Mar 2011 15:38:44 1080604 11088983 hadsm3dhet2_junn_006602725_4 205,238 1,205,333 1.6654
28 Feb 2011 08:10:37 1080604 11088983 hadsm3dhet2_junn_006602725_4 194,436 1,188,858 1.6676
23 Feb 2011 14:46:29 1080604 11088983 hadsm3dhet2_junn_006602725_4 183,634 1,172,395 1.6698
21 Feb 2011 00:46:22 1080604 11088983 hadsm3dhet2_junn_006602725_4 172,832 1,155,737 1.6718
19 Feb 2011 14:12:50 1080604 11088983 hadsm3dhet2_junn_006602725_4 162,030 1,138,868 1.6735
16 Feb 2011 05:18:26 1080604 11088983 hadsm3dhet2_junn_006602725_4 151,228 1,122,166 1.6756
12 Feb 2011 05:52:16 1080604 11088983 hadsm3dhet2_junn_006602725_4 140,426 1,105,421 1.6776
06 Feb 2011 11:02:20 1080604 11088983 hadsm3dhet2_junn_006602725_4 129,624 1,088,848 1.6800
02 Feb 2011 20:11:59 1080604 11088983 hadsm3dhet2_junn_006602725_4 118,822 1,072,517 1.6829
30 Jan 2011 08:04:55 1080604 11088983 hadsm3dhet2_junn_006602725_4 108,020 1,056,032 1.6856
27 Jan 2011 11:08:23 1080604 11088983 hadsm3dhet2_junn_006602725_4 97,218 1,039,569 1.6884
12 Jan 2011 16:13:30 1080604 11088983 hadsm3dhet2_junn_006602725_4 86,416 1,023,059 1.6913
11 Jan 2011 23:38:05 1080604 11088983 hadsm3dhet2_junn_006602725_4 75,614 1,006,473 1.6941
11 Jan 2011 00:35:18 1080604 11088983 hadsm3dhet2_junn_006602725_4 64,812 989,440 1.6963
10 Jan 2011 13:27:56 1080604 11088983 hadsm3dhet2_junn_006602725_4 54,010 972,420 1.6985
07 Jan 2011 11:30:50 1080604 11088983 hadsm3dhet2_junn_006602725_4 43,208 955,377 1.7009
06 Jan 2011 16:51:36 1080604 11088983 hadsm3dhet2_junn_006602725_4 32,406 936,957 1.7008
03 Jan 2011 07:30:13 1080604 11088983 hadsm3dhet2_junn_006602725_4 21,604 919,244 1.7020
01 Jan 2011 09:48:54 1080604 11088983 hadsm3dhet2_junn_006602725_4 10,802 900,974 1.7022
30 Dec 2010 03:43:09 1080604 11088983 hadsm3dhet2_junn_006602725_4 259,248 882,526 1.7021


©2024 cpdn.org