climateprediction.net home page
Task 19451708

Task 19451708

Name hadcm3n_sdo4_198012_480_375_010414470_2
Workunit 10414470
Created 24 Mar 2016, 7:58:30 UTC
Sent 24 Mar 2016, 7:59:25 UTC
Report deadline 6 Mar 2017, 13:19:25 UTC
Received 25 May 2016, 3:17:06 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1319963
Run time 37 days 22 hours 35 min 55 sec
CPU time 34 days 9 hours 14 min 31 sec
Validate state Invalid
Credit 9,331.20
Device peak FLOPS 1.38 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr 
<core_client_version>7.6.22</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
22:10:05 (2318): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
19:02:51 (614): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
23:12:16 (18179): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
19:49:34 (60208): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
20:54:40 (68901): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
21:38:37 (69414): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:35:36 (69449): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
20:07:43 (87372): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:52:53 (96082): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
15:39:37 (2455): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
00:31:17 (3252): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
14:16:35 (29825): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
20:23:06 (640): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
20:05:21 (2878): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
20:10:14 (48087): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
16:13:11 (24161): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:23:25 (30063): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:23:27 (30063): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - Suspend request from BOINC...
20:08:36 (31030): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
23:41:33 (39737): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:41:34 (39737): No heartbeat from core client for 30 sec - exiting
23:41:35 (39737): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
15:19:25 (12317): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
21:47:09 (29508): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:33:56 (33535): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
00:34:04 (34243): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:04:08 (34636): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
hadcm3n_6.07_i686-apple-darwin(34843,0xa435e000) malloc: *** error for object 0x4801c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(34843,0xa435e000) malloc: *** error for object 0x4801c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(34843,0xa435e000) malloc: *** error for object 0x3001c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(34843,0xa435e000) malloc: *** error for object 0x3001c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(34843,0xa435e000) malloc: *** error for object 0x3001c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation
hadcm3n_6.07_i686-apple-darwin(34843,0xa435e000) malloc: *** error for object 0x808e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug

Crashed executable name: hadcm3n_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.11.5 build 15F34
Mon May 23 19:53:10 2016

hadcm3n_6.07_i686-apple-darwin(34843,0xa435e000) malloc: *** error for object 0x80ce04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Thread 0 Crashed:
atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386.
hadcm3n_6.07_i686-apple-darwin(34843,0xa435e000) malloc: *** error for object 0x80e004: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(34843,0xa435e000) malloc: *** error for object 0x80e000: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug

Thread 1:
20:24:58 (34843): No heartbeat from core client for 30 sec - exiting
20:24:59 (34843): No heartbeat from core client for 30 sec - exiting
20:25:00 (34843): No heartbeat from core client for 30 sec - exiting
20:25:01 (34843): No heartbeat from core client for 30 sec - exiting
20:25:02 (34843): No heartbeat from core client for 30 sec - exiting
20:25:03 (34843): No heartbeat from core client for 30 sec - exiting
20:25:04 (34843): No heartbeat from core client for 30 sec - exiting
20:25:05 (34843): No heartbeat from core client for 30 sec - exiting
20:25:06 (34843): No heartbeat from core client for 30 sec - exiting
20:25:07 (34843): No heartbeat from core client for 30 sec - exiting
20:25:08 (34843): No heartbeat from core client for 30 sec - exiting
20:25:09 (34843): No heartbeat from core client for 30 sec - exiting
20:25:10 (34843): No heartbeat from core client for 30 sec - exiting
20:25:11 (34843): No heartbeat from core client for 30 sec - exiting
20:25:12 (34843): No heartbeat from core client for 30 sec - exiting
20:25:13 (34843): No heartbeat from core client for 30 sec - exiting
20:25:14 (34843): No heartbeat from core client for 30 sec - exiting
20:25:15 (34843): No heartbeat from core client for 30 sec - exiting
20:25:16 (34843): No heartbeat from core client for 30 sec - exiting
20:25:17 (34843): No heartbeat from core client for 30 sec - exiting
20:25:19 (34843): No heartbeat from core client for 30 sec - exiting
20:25:20 (34843): No heartbeat from core client for 30 sec - exiting
20:25:21 (34843): No heartbeat from core client for 30 sec - exiting
20:25:22 (34843): No heartbeat from core client for 30 sec - exiting
20:25:23 (34843): No heartbeat from core client for 30 sec - exiting
20:25:24 (34843): No heartbeat from core client for 30 sec - exiting
20:25:25 (34843): No heartbeat from core client for 30 sec - exiting
20:25:26 (34843): No heartbeat from core client for 30 sec - exiting
20:25:27 (34843): No heartbeat from core client for 30 sec - exiting
20:25:28 (34843): No heartbeat from core client for 30 sec - exiting
hadcm3n_6.07_i686-apple-darwin(43095,0xa435e000) malloc: *** error for object 0x2009e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43095,0xa435e000) malloc: *** error for object 0x2009e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43095,0xa435e000) malloc: *** error for object 0x2009e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43095,0xa435e000) malloc: *** error for object 0x2009e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43095,0xa435e000) malloc: *** error for object 0x2009e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation
hadcm3n_6.07_i686-apple-darwin(45679,0xa435e000) malloc: *** error for object 0x4009c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(45679,0xa435e000) malloc: *** error for object 0x4009c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(45679,0xa435e000) malloc: *** error for object 0x2801c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(45679,0xa435e000) malloc: *** error for object 0x2801c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(45679,0xa435e000) malloc: *** error for object 0x2801c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation
hadcm3n_6.07_i686-apple-darwin(45679,0xb0005000) malloc: *** error for object 0x5000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(45679,0xb0005000) malloc: *** error for object 0x5001204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
20:10:39 (45679): No heartbeat from core client for 30 sec - exiting
20:10:40 (45679): No heartbeat from core client for 30 sec - exiting
20:10:41 (45679): No heartbeat from core client for 30 sec - exiting
20:10:42 (45679): No heartbeat from core client for 30 sec - exiting
20:10:43 (45679): No heartbeat from core client for 30 sec - exiting
20:10:44 (45679): No heartbeat from core client for 30 sec - exiting
20:10:45 (45679): No heartbeat from core client for 30 sec - exiting
20:10:46 (45679): No heartbeat from core client for 30 sec - exiting
20:10:47 (45679): No heartbeat from core client for 30 sec - exiting
20:10:48 (45679): No heartbeat from core client for 30 sec - exiting
20:10:49 (45679): No heartbeat from core client for 30 sec - exiting
20:10:51 (45679): No heartbeat from core client for 30 sec - exiting
20:10:52 (45679): No heartbeat from core client for 30 sec - exiting
20:10:53 (45679): No heartbeat from core client for 30 sec - exiting
20:10:54 (45679): No heartbeat from core client for 30 sec - exiting
20:10:55 (45679): No heartbeat from core client for 30 sec - exiting
20:10:56 (45679): No heartbeat from core client for 30 sec - exiting
20:10:57 (45679): No heartbeat from core client for 30 sec - exiting
20:10:58 (45679): No heartbeat from core client for 30 sec - exiting
20:10:59 (45679): No heartbeat from core client for 30 sec - exiting
20:11:00 (45679): No heartbeat from core client for 30 sec - exiting
20:11:01 (45679): No heartbeat from core client for 30 sec - exiting
20:11:02 (45679): No heartbeat from core client for 30 sec - exiting
20:11:03 (45679): No heartbeat from core client for 30 sec - exiting
20:11:04 (45679): No heartbeat from core client for 30 sec - exiting
20:11:05 (45679): No heartbeat from core client for 30 sec - exiting
20:11:06 (45679): No heartbeat from core client for 30 sec - exiting
20:11:07 (45679): No heartbeat from core client for 30 sec - exiting
20:11:08 (45679): No heartbeat from core client for 30 sec - exiting
20:11:09 (45679): No heartbeat from core client for 30 sec - exiting
20:11:10 (45679): No heartbeat from core client for 30 sec - exiting
hadcm3n_6.07_i686-apple-darwin(50967,0xa435e000) malloc: *** error for object 0x804e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(50967,0xa435e000) malloc: *** error for object 0x804e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(50967,0xa435e000) malloc: *** error for object 0x1019e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(50967,0xa435e000) malloc: *** error for object 0x1019e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(50967,0xa435e000) malloc: *** error for object 0x5001c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation

Crashed executable name: hadcm3n_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.11.5 build 15F34
Tue May 24 22:16:43 2016

hadcm3n_6.07_i686-apple-darwin(50967,0xa435e000) malloc: *** error for object 0x1004e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(50967,0xa435e000) malloc: *** error for object 0x1004e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Thread 0 Crashed:

Thread 1:

atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386.
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fda8 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -    0x93fff /Library/Application Support/BOINC Data/slots/8/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin
0x903ec000 - 0x903effff /usr/lib/system/libsystem_sandbox.dylib
0x90bc2000 - 0x90bdbfff /usr/lib/system/libsystem_asl.dylib
0x90f60000 - 0x90f62fff /usr/lib/system/libsystem_configuration.dylib
0x916ef000 - 0x91717fff /usr/lib/system/libxpc.dylib
0x91718000 - 0x91718fff /usr/lib/system/libsystem_blocks.dylib
0x918bd000 - 0x918befff /usr/lib/system/libremovefile.dylib
0x94001000 - 0x9400afff /usr/lib/system/libsystem_networkextension.dylib
0x9436a000 - 0x94374fff /usr/lib/system/libsystem_notify.dylib
0x94ccb000 - 0x94ce4fff /usr/lib/system/libsystem_malloc.dylib
0x94d19000 - 0x94d24fff /usr/lib/system/libcommonCrypto.dylib
0x95807000 - 0x95808fff /usr/lib/libSystem.B.dylib
0x9580c000 - 0x95836fff /usr/lib/system/libsystem_info.dylib
0x9708c000 - 0x9708efff /usr/lib/system/libsystem_secinit.dylib
0x97bba000 - 0x97bdbfff /usr/lib/system/libsystem_kernel.dylib
0x97bdc000 - 0x97bdcfff /usr/lib/system/liblaunch.dylib
0x97d3e000 - 0x97d68fff /usr/lib/system/libdispatch.dylib
0x97d69000 - 0x97d6cfff /usr/lib/system/libdyld.dylib
0x97d6d000 - 0x97d71fff /usr/lib/system/libcache.dylib
0x98d0e000 - 0x98d75fff /usr/lib/system/libcorecrypto.dylib
0x98dfd000 - 0x98e20fff /usr/lib/libc++abi.dylib
0x98e70000 - 0x98e78fff /usr/lib/system/libsystem_dnssd.dylib
0x994c4000 - 0x994f7fff /usr/lib/system/libsystem_m.dylib
0x99504000 - 0x99515fff /usr/lib/system/libsystem_trace.dylib
0x99764000 - 0x99766fff /usr/lib/system/libsystem_coreservices.dylib
0x99a2f000 - 0x99a3dfff /usr/lib/libz.1.dylib
0x99a3e000 - 0x99a44fff /usr/lib/system/libsystem_platform.dylib
0x9ac4f000 - 0x9ac66fff /usr/lib/system/libsystem_coretls.dylib
0x9acb3000 - 0x9acb9fff /usr/lib/system/libunwind.dylib
0x9acba000 - 0x9acbcfff /usr/lib/system/libquarantine.dylib
0x9acbd000 - 0x9ad2bfff /usr/lib/system/libsystem_network.dylib
0x9ad2c000 - 0x9ad7ffff /usr/lib/libstdc++.6.dylib
0x9bd35000 - 0x9bd3afff /usr/lib/system/libmacho.dylib
0x9c42f000 - 0x9c4c4fff /usr/lib/system/libsystem_c.dylib
0x9d3f0000 - 0x9d3f8fff /usr/lib/system/libsystem_pthread.dylib
0x9db89000 - 0x9db8efff /usr/lib/system/libcompiler_rt.dylib
0x9df95000 - 0x9df9efff /usr/lib/system/libcopyfile.dylib
0x9e26b000 - 0x9e26bfff /usr/lib/system/libkeymgr.dylib
0x9e2c9000 - 0x9e2c9fff /usr/lib/system/libunc.dylib


Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
23 May 2016 23:59:10 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 777,600 2,969,573 3.8189
22 May 2016 18:29:43 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 751,680 2,872,900 3.8220
21 May 2016 07:23:08 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 725,760 2,776,417 3.8255
19 May 2016 18:04:09 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 699,840 2,679,521 3.8288
16 May 2016 14:17:44 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 673,920 2,581,578 3.8307
15 May 2016 07:02:11 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 648,000 2,482,545 3.8311
13 May 2016 20:37:09 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 622,080 2,383,333 3.8312
11 May 2016 18:59:41 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 596,160 2,284,143 3.8314
10 May 2016 07:12:10 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 570,240 2,185,292 3.8322
07 May 2016 22:58:39 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 544,320 2,085,972 3.8323
06 May 2016 14:17:39 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 518,400 1,987,322 3.8336
05 May 2016 09:26:07 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 492,480 1,888,711 3.8351
30 Apr 2016 17:39:02 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 466,560 1,790,057 3.8367
29 Apr 2016 01:10:23 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 440,640 1,690,069 3.8355
25 Apr 2016 17:04:40 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 414,720 1,589,932 3.8337
24 Apr 2016 09:54:09 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 388,800 1,490,148 3.8327
21 Apr 2016 19:24:20 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 362,880 1,390,572 3.8320
19 Apr 2016 09:00:12 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 336,960 1,290,643 3.8303
18 Apr 2016 02:51:43 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 311,040 1,190,286 3.8268
16 Apr 2016 00:55:57 1319963 19451708 hadcm3n_sdo4_198012_480_375_010414470_2 285,120 1,090,201 3.8237


©2024 cpdn.org