climateprediction.net home page
Task 11367855

Task 11367855

Name hadsm3dhet2_kg67_006630609_9
Workunit 6833982
Created 15 Mar 2010, 12:46:55 UTC
Sent 18 Mar 2010, 5:05:56 UTC
Report deadline 28 Feb 2011, 10:25:56 UTC
Received 27 Aug 2010, 20:48:09 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 890735
Run time
CPU time 43 days 0 hours 7 min 44 sec
Validate state Invalid
Credit 6,550.06
Device peak FLOPS 0.00 GFLOPS
Application version ---
Stderr
<core_client_version>5.10.45</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=15397, selfPID=15397, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
MainError:	06:14:21 PM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: 7R
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
MainError:	06:13:59 PM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12144, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12144, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: 7R
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22670, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: 7R

Model crashed: 7R
CPDN process is not running, exiting, bRetVal = 1, checkPID=25720, selfPID=25720, iMonCtr=1

Model crashed: 7R
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3420, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: 7R

Model crashed: 7R
CPDN Monitor - Quit request from BOINC...

Model crashed: 7R
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15884, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: 7R

Model crashed: 7R

Model crashed: 7R
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x8496080]
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x833b396]
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x8250189]
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x811840c]
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x80b159f]
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x86c8b1e]
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x86f5990]
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x86c0886]
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x86feb0f]
/lib/libc.so.6(__libc_start_main+0xdc)[0x559e8c]
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu(pthread_mutex_unlock+0x111)[0x804bb31]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15884, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: 7R
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15884, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15884, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15884, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: 7R

Model crashed: 7R
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=28411, selfPID=28411, iMonCtr=1
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10856, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11442, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11442, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=14962, selfPID=14962, iMonCtr=1
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12362, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12362, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12362, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...

Model crashed: 7R
SIGSEGV: segmentation violation
Stack trace (4 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x85f93dc]
[0x80000000]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24856, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: 7R
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: 7R
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation
Stack trace (2 frames):
/home/seti/BOINC/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu[0x876060b]
[0x40000420]

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23636, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: 7R
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: 7R
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: 7R

Model crashed: 7R

Model crashed: 7R

Model crashed: 7R

Model crashed: 7R

Model crashed: 7R
Sorry, too many model crashes! :-(
called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
26 Aug 2010 19:04:12 890735 11367855 hadsm3dhet2_kg67_006630609_9 194,436 3,677,271 5.1580
25 Aug 2010 02:02:26 890735 11367855 hadsm3dhet2_kg67_006630609_9 183,634 3,623,982 5.1614
23 Aug 2010 18:16:40 890735 11367855 hadsm3dhet2_kg67_006630609_9 172,832 3,568,327 5.1616
22 Aug 2010 05:58:32 890735 11367855 hadsm3dhet2_kg67_006630609_9 162,030 3,513,332 5.1627
21 Aug 2010 01:35:18 890735 11367855 hadsm3dhet2_kg67_006630609_9 151,228 3,456,539 5.1611
19 Aug 2010 23:41:36 890735 11367855 hadsm3dhet2_kg67_006630609_9 140,426 3,399,370 5.1590
18 Aug 2010 19:11:26 890735 11367855 hadsm3dhet2_kg67_006630609_9 129,624 3,349,779 5.1685
14 Jul 2010 19:58:05 890735 11367855 hadsm3dhet2_kg67_006630609_9 118,822 3,297,731 5.1744
13 Jul 2010 20:07:12 890735 11367855 hadsm3dhet2_kg67_006630609_9 108,020 3,245,140 5.1797
10 Jul 2010 00:19:57 890735 11367855 hadsm3dhet2_kg67_006630609_9 97,218 3,193,185 5.1861
08 Jul 2010 02:14:00 890735 11367855 hadsm3dhet2_kg67_006630609_9 86,416 3,140,180 5.1911
06 Jul 2010 22:52:40 890735 11367855 hadsm3dhet2_kg67_006630609_9 75,614 3,078,207 5.1812
05 Jul 2010 20:23:49 890735 11367855 hadsm3dhet2_kg67_006630609_9 64,812 3,016,086 5.1707
01 Jul 2010 17:57:57 890735 11367855 hadsm3dhet2_kg67_006630609_9 54,010 2,953,597 5.1591
30 Jun 2010 15:17:00 890735 11367855 hadsm3dhet2_kg67_006630609_9 43,208 2,897,303 5.1581
29 Jun 2010 03:04:37 890735 11367855 hadsm3dhet2_kg67_006630609_9 32,406 2,833,276 5.1430
28 Jun 2010 00:05:04 890735 11367855 hadsm3dhet2_kg67_006630609_9 21,604 2,768,891 5.1266
26 Jun 2010 21:01:54 890735 11367855 hadsm3dhet2_kg67_006630609_9 10,802 2,704,850 5.1103
25 Jun 2010 18:16:24 890735 11367855 hadsm3dhet2_kg67_006630609_9 259,248 2,641,259 5.0941
24 Jun 2010 05:55:18 890735 11367855 hadsm3dhet2_kg67_006630609_9 248,446 2,577,311 5.0765


©2024 climateprediction.net