climateprediction.net home page
Task 16548846

Task 16548846

Name hadam3pm2_c966_1975_10_008682073_1
Workunit 8837403
Created 18 Apr 2014, 20:46:03 UTC
Sent 19 Apr 2014, 1:50:00 UTC
Report deadline 1 Apr 2015, 7:10:00 UTC
Received 7 Jun 2014, 8:10:05 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1297353
Run time 11 days 21 hours 43 min 34 sec
CPU time 11 days 12 hours 58 min 20 sec
Validate state Invalid
Credit 9,066.40
Device peak FLOPS 2.49 GFLOPS
Application version UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03
i686-pc-linux-gnu
Stderr
<core_client_version>7.1.0</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
[0xf0f9c400]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/usr/lib/libc.so.6(__libc_start_main+0xf3)[0xf0d92963]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12283, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
[0xf0fa6400]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/usr/lib/libc.so.6(__libc_start_main+0xf3)[0xf0d9c963]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12283, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
[0xf0eff400]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/usr/lib/libc.so.6(__libc_start_main+0xf3)[0xf0cf5963]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12283, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
[0xf0f78400]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/usr/lib/libc.so.6(__libc_start_main+0xf3)[0xf0d6e963]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12283, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
[0xf0f6b400]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/usr/lib/libc.so.6(__libc_start_main+0xf3)[0xf0d61963]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12283, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
[0xf0f62400]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/usr/local/ondrejch/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/usr/lib/libc.so.6(__libc_start_main+0xf3)[0xf0d58963]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12283, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
03:36:47 (12283): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Jun 2014 13:58:23 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 313,988 1,009,548 3.2152
04 Jun 2014 12:26:39 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 288,068 915,593 3.1784
04 Jun 2014 09:41:13 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 285,188 905,602 3.1755
04 Jun 2014 06:55:33 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 282,308 895,900 3.1735
04 Jun 2014 04:10:05 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 279,428 886,001 3.1708
01 Jun 2014 16:37:32 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 244,868 760,755 3.1068
29 May 2014 21:59:42 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 210,308 634,651 3.0177
27 May 2014 15:41:18 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 175,748 559,734 3.1849
02 May 2014 18:12:55 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 141,188 478,855 3.3916
24 Apr 2014 22:55:59 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 109,508 371,840 3.3956
24 Apr 2014 20:15:17 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 106,628 362,200 3.3969
22 Apr 2014 17:41:48 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 74,948 255,568 3.4099
22 Apr 2014 14:53:10 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 72,068 245,396 3.4051
20 Apr 2014 23:03:13 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 37,508 125,755 3.3528
19 Apr 2014 04:58:48 1297353 16548846 hadam3pm2_c966_1975_10_008682073_1 2,948 9,584 3.2510


©2024 climateprediction.net