climateprediction.net home page
Task 19129798

Task 19129798

Name hadcm3n_xa09_198012_480_010218839_0
Workunit 10218839
Created 4 Dec 2015, 14:04:36 UTC
Sent 4 Dec 2015, 14:11:44 UTC
Report deadline 15 Nov 2016, 19:31:44 UTC
Received 25 Dec 2015, 6:57:00 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1378735
Run time 20 days 3 hours 54 min 13 sec
CPU time 19 days 7 hours 12 min 15 sec
Validate state Invalid
Credit 11,819.52
Device peak FLOPS 2.19 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>7.0.27</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
17:08:54 (1154): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:21:34 (5865): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:24:00 (6461): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:25:52 (6481): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:27:49 (6501): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:29:40 (6521): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:32:11 (6541): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:32:12 (6541): No heartbeat from core client for 30 sec - exiting
19:32:14 (6541): No heartbeat from core client for 30 sec - exiting
19:32:15 (6541): No heartbeat from core client for 30 sec - exiting
19:34:28 (6561): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:37:10 (6581): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:39:16 (6601): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:41:07 (6621): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:43:05 (6641): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:45:30 (6661): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:45:31 (6661): No heartbeat from core client for 30 sec - exiting
19:45:32 (6661): No heartbeat from core client for 30 sec - exiting
19:45:34 (6661): No heartbeat from core client for 30 sec - exiting
19:45:35 (6661): No heartbeat from core client for 30 sec - exiting
19:47:42 (6681): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:49:48 (6701): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:51:59 (6721): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:54:00 (6741): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:56:37 (6761): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:58:43 (6781): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:01:14 (6801): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:01:15 (6801): No heartbeat from core client for 30 sec - exiting
20:01:16 (6801): No heartbeat from core client for 30 sec - exiting
20:01:17 (6801): No heartbeat from core client for 30 sec - exiting
20:03:55 (6821): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:05:51 (6841): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:08:27 (6861): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:10:33 (6881): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:12:45 (6901): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:15:06 (6921): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:16:57 (6941): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:18:54 (6964): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:20:45 (6984): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:23:16 (7004): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:23:17 (7004): No heartbeat from core client for 30 sec - exiting
20:23:18 (7004): No heartbeat from core client for 30 sec - exiting
20:23:19 (7004): No heartbeat from core client for 30 sec - exiting
20:25:27 (7024): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:28:03 (7044): No heartbeat from core client for 30 sec - exiting
20:28:04 (7044): No heartbeat from core client for 30 sec - exiting
20:28:05 (7044): No heartbeat from core client for 30 sec - exiting
20:28:06 (7044): No heartbeat from core client for 30 sec - exiting
20:28:07 (7044): No heartbeat from core client for 30 sec - exiting
20:28:08 (7044): No heartbeat from core client for 30 sec - exiting
20:28:09 (7044): No heartbeat from core client for 30 sec - exiting
20:28:10 (7044): No heartbeat from core client for 30 sec - exiting
20:28:11 (7044): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Error: Input file: dataout/xa09ko.pjl8c10 is not a valid UM file.
Error converting file to netcdf: dataout/xa09ko.pjl8c10
Error: Input file: dataout/xa09ko.pil8c10 is not a valid UM file.
Error converting file to netcdf: dataout/xa09ko.pil8c10
Error: Input file: dataout/xa09ko.pfl8c10 is not a valid UM file.
Error converting file to netcdf: dataout/xa09ko.pfl8c10
Error: Input file: dataout/xa09ka.phl8c10 is not a valid UM file.
Error converting file to netcdf: dataout/xa09ka.phl8c10
Error: Input file: dataout/xa09ka.pgl8c10 is not a valid UM file.
Error converting file to netcdf: dataout/xa09ka.pgl8c10
Error: Input file: dataout/xa09ka.pel8c10 is not a valid UM file.
Error converting file to netcdf: dataout/xa09ka.pel8c10
Error: Input file: dataout/xa09ka.pdl8c10 is not a valid UM file.
Error converting file to netcdf: dataout/xa09ka.pdl8c10
20:30:20 (7064): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
20:32:10 (7084): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
20:33:56 (7104): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
20:36:22 (7124): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:38:23 (7144): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:40:59 (7164): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:43:01 (7184): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:45:02 (7204): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:47:03 (7224): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:49:14 (7244): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:51:30 (7264): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:54:11 (7284): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:56:47 (7304): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:59:23 (7324): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:01:34 (7344): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:03:32 (7364): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:06:03 (7384): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:08:35 (7404): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:11:16 (7424): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:13:42 (7444): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:15:29 (7464): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:18:05 (7484): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:19:56 (7507): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:22:07 (7527): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:24:23 (7547): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:26:24 (7567): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:28:46 (7587): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:28:47 (7587): No heartbeat from core client for 30 sec - exiting
21:31:12 (7607): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:31:13 (7607): No heartbeat from core client for 30 sec - exiting
21:31:14 (7607): No heartbeat from core client for 30 sec - exiting
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xb773e400]
[0xb773e424]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb756d1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7570825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75584d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7627, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xb77a1400]
[0xb77a1424]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75d01df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb75d3825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75bb4d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7627, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xb76ff400]
[0xb76ff424]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb752e1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7531825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75194d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7627, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xb77b1400]
[0xb77b1424]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75e01df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb75e3825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75cb4d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7627, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xb77cf400]
[0xb77cf424]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75fe1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7601825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75e94d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7627, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xb7721400]
[0xb7721424]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75501df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7553825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb753b4d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7627, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
25 Dec 2015 06:56:57 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 984,960 1,667,552 1.6930
24 Dec 2015 06:19:49 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 959,040 1,623,836 1.6932
23 Dec 2015 17:17:19 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 933,120 1,578,953 1.6921
23 Dec 2015 04:12:53 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 907,200 1,533,983 1.6909
22 Dec 2015 15:09:51 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 881,280 1,489,046 1.6896
22 Dec 2015 02:09:08 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 855,360 1,444,147 1.6883
21 Dec 2015 13:09:23 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 829,440 1,399,345 1.6871
21 Dec 2015 00:06:05 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 803,520 1,354,579 1.6858
20 Dec 2015 11:18:44 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 777,600 1,310,359 1.6851
19 Dec 2015 22:24:36 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 751,680 1,266,208 1.6845
19 Dec 2015 09:42:04 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 725,760 1,222,202 1.6840
18 Dec 2015 20:55:14 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 699,840 1,178,333 1.6837
18 Dec 2015 08:17:31 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 673,920 1,134,469 1.6834
17 Dec 2015 19:32:37 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 648,000 1,090,703 1.6832
17 Dec 2015 06:49:08 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 622,080 1,046,992 1.6831
16 Dec 2015 17:41:49 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 596,160 1,003,290 1.6829
16 Dec 2015 05:27:31 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 570,240 959,571 1.6827
15 Dec 2015 16:10:50 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 544,320 915,762 1.6824
15 Dec 2015 04:00:32 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 518,400 872,060 1.6822
14 Dec 2015 14:58:01 1378735 19129798 hadcm3n_xa09_198012_480_010218839_0 492,480 829,491 1.6843


©2024 cpdn.org