climateprediction.net home page
Task 13741205

Task 13741205

Name hadcm3n_004y_1940_40_007607663_2
Workunit 7785793
Created 7 Dec 2011, 1:49:26 UTC
Sent 7 Dec 2011, 2:42:02 UTC
Report deadline 7 Mar 2012, 10:09:13 UTC
Received 28 Dec 2011, 6:27:35 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1189010
Run time 13 days 3 hours 21 min 12 sec
CPU time 12 days 23 hours 45 min 29 sec
Validate state Invalid
Credit 9,331.20
Device peak FLOPS 3.03 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr 
<core_client_version>6.12.34</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
18:04:10 (9756): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:06:31 (9849): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:08:22 (10016): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:10:54 (10144): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:12:40 (10311): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:14:56 (10469): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:17:01 (10586): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:41:12 (10736): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:46:23 (22207): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:52:04 (22240): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:00:05 (22289): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:15:02 (22687): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:16:53 (28986): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:26:48 (29017): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:40:44 (29055): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:42:21 (29100): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:54:24 (29126): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29212, iMonCtr=1
Model crash detected, will try to restart...
Signal 15 received, exiting...
Called boinc_finish
SIGSEGV: segmentation violation
Stack trace (7 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xf7702400]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xe7)[0xf744ee37]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...
11:46:04 (3197): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:51:20 (5282): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:58:17 (5326): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:00:43 (5362): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:30:37 (5393): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:38:33 (7861): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:05:21 (8629): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:07:58 (8704): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:22:30 (8737): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:35:22 (8793): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:04:12 (8850): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:46:55 (8915): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:48:52 (25515): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:19:02 (25544): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:21:40 (25840): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:23:36 (25870): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:26:02 (25899): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:28:14 (25934): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:32:38 (25965): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:37:36 (25999): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:39:58 (26037): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:51:30 (26069): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:53:26 (26129): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:13:08 (26164): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:15:10 (27240): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:17:16 (27269): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:19:34 (27307): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:22:00 (27339): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:25:51 (27370): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:30:01 (27404): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Signal 15 received, exiting...
Called boinc_finish
09:49:37 (2646): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
09:56:58 (3689): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:14:54 (3726): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:17:31 (3780): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:31:40 (3816): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:00:06 (3864): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:03:20 (3928): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:05:28 (3961): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:29:06 (3990): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:09:36 (4054): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:14:05 (4256): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:14:37 (4329): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:17:31 (4371): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:46:50 (4579): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:48:10 (9764): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:44:53 (9792): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:46:27 (14701): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:52:05 (14729): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:53:42 (14762): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:55:08 (14801): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:56:11 (14834): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:57:48 (14863): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:58:25 (14891): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:59:12 (14919): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:02:26 (14948): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
12:44:38 (14979): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
22:40:42 (24925): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
22:45:25 (32606): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:23:39 (32628): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:25:26 (1081): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:14:45 (1098): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:15:57 (1221): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:16:43 (1236): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:19:11 (1250): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:21:17 (1280): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:23:54 (1296): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:25:48 (1314): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:29:35 (1334): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:32:07 (1354): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:34:14 (1374): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:36:11 (1392): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:35:46 (1407): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:47:53 (27911): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Signal 15 received, exiting...
Called boinc_finish
23:09:49 (2280): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:55:27 (5924): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:56:08 (6265): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:12:41 (6273): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:14:02 (7485): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:18:00 (7489): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:18:40 (7512): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:14:07 (7519): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:20:36 (7672): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08a64d08 ***
======= Backtrace: =========
/lib32/libc.so.6(+0x6b511)[0xf7515511]
/lib32/libc.so.6(+0x6ce1b)[0xf7516e1b]
/lib32/libc.so.6(cfree+0x6d)[0xf7519f8d]
/usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf76f3841]
/usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf76f389d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xe7)[0xf74c0e37]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]
======= Memory map: ========
08048000-080e3000 r-xp 00000000 08:01 16647274                           /usr/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e3000-080e4000 rw-p 0009b000 08:01 16647274                           /usr/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e4000-0813b000 rw-p 00000000 00:00 0 
08a0f000-08a74000 rw-p 00000000 00:00 0                                  [heap]
f6f00000-f6f21000 rw-p 00000000 00:00 0 
f6f21000-f7000000 ---p 00000000 00:00 0 
f702c000-f74a7000 rw-s 00000000 08:01 16777430                           /usr/BOINC/slots/6/132335
f74a7000-f74aa000 rw-p 00000000 00:00 0 
f74aa000-f7600000 r-xp 00000000 08:01 16646149                           /lib32/libc-2.13.so
f7600000-f7601000 ---p 00156000 08:01 16646149                           /lib32/libc-2.13.so
f7601000-f7603000 r--p 00156000 08:01 16646149                           /lib32/libc-2.13.so
f7603000-f7604000 rw-p 00158000 08:01 16646149                           /lib32/libc-2.13.so
f7604000-f7607000 rw-p 00000000 00:00 0 
f7607000-f7621000 r-xp 00000000 08:01 12193083                           /usr/lib32/libgcc_s.so.1
f7621000-f7622000 r--p 00019000 08:01 12193083                           /usr/lib32/libgcc_s.so.1
f7622000-f7623000 rw-p 0001a000 08:01 12193083                           /usr/lib32/libgcc_s.so.1
f7623000-f7647000 r-xp 00000000 08:01 16646153                           /lib32/libm-2.13.so
f7647000-f7648000 r--p 00023000 08:01 16646153                           /lib32/libm-2.13.so
f7648000-f7649000 rw-p 00024000 08:01 16646153                           /lib32/libm-2.13.so
f7649000-f7728000 r-xp 00000000 08:01 12193086                           /usr/lib32/libstdc++.so.6.0.14
f7728000-f772c000 r--p 000de000 08:01 12193086                           /usr/lib32/libstdc++.so.6.0.14
f772c000-f772d000 rw-p 000e2000 08:01 12193086                           /usr/lib32/libstdc++.so.6.0.14
f772d000-f7734000 rw-p 00000000 00:00 0 
f7734000-f7736000 r-xp 00000000 08:01 16646152                           /lib32/libdl-2.13.so
f7736000-f7737000 r--p 00001000 08:01 16646152                           /lib32/libdl-2.13.so
f7737000-f7738000 rw-p 00002000 08:01 16646152                           /lib32/libdl-2.13.so
f7738000-f774d000 r-xp 00000000 08:01 16646163                           /lib32/libpthread-2.13.so
f774d000-f774e000 r--p 00014000 08:01 16646163                           /lib32/libpthread-2.13.so
f774e000-f774f000 rw-p 00015000 08:01 16646163                           /lib32/libpthread-2.13.so
f774f000-f7752000 rw-p 00000000 00:00 0 
f776c000-f776d000 rw-p 00000000 00:00 0 
f776d000-f776e000 ---p 00000000 00:00 0 
f776e000-f7771000 rw-p 00000000 00:00 0 
f7771000-f7773000 rw-s 00000000 08:01 16777427                           /usr/BOINC/slots/6/boinc_mmap_file
f7773000-f7774000 rw-p 00000000 00:00 0 
f7774000-f7775000 r-xp 00000000 00:00 0                                  [vdso]
f7775000-f7791000 r-xp 00000000 08:01 16646146                           /lib32/ld-2.13.so
f7791000-f7792000 r--p 0001b000 08:01 16646146                           /lib32/ld-2.13.so
f7792000-f7793000 rw-p 0001c000 08:01 16646146                           /lib32/ld-2.13.so
ff8fe000-ff96e000 rw-p 00000000 00:00 0                                  [stack]
SIGABRT: abort called
Stack trace (19 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xf7774400]
[0xf7774430]
/lib32/libc.so.6(gsignal+0x51)[0xf74d4ea1]
/lib32/libc.so.6(abort+0x17e)[0xf74d82ce]
/lib32/libc.so.6(+0x61277)[0xf750b277]
/lib32/libc.so.6(+0x6b511)[0xf7515511]
/lib32/libc.so.6(+0x6ce1b)[0xf7516e1b]
/lib32/libc.so.6(cfree+0x6d)[0xf7519f8d]
/usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf76f3841]
/usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf76f389d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xe7)[0xf74c0e37]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
28 Dec 2011 06:29:16 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 777,600 1,122,325 1.4433
27 Dec 2011 21:16:52 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 751,680 1,090,108 1.4502
27 Dec 2011 12:08:18 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 725,760 1,057,719 1.4574
27 Dec 2011 02:56:08 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 699,840 1,025,360 1.4651
26 Dec 2011 17:50:53 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 673,920 993,014 1.4735
26 Dec 2011 08:45:07 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 648,000 960,644 1.4825
25 Dec 2011 23:33:18 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 622,080 927,938 1.4917
25 Dec 2011 14:20:02 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 596,160 895,013 1.5013
25 Dec 2011 05:03:03 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 570,240 862,174 1.5119
24 Dec 2011 19:49:55 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 544,320 829,323 1.5236
24 Dec 2011 10:40:16 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 518,400 796,491 1.5364
24 Dec 2011 01:28:34 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 492,480 763,725 1.5508
23 Dec 2011 16:07:47 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 466,560 730,454 1.5656
23 Dec 2011 07:10:54 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 440,640 697,136 1.5821
22 Dec 2011 21:12:39 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 414,720 664,322 1.6019
22 Dec 2011 12:24:30 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 388,800 631,072 1.6231
22 Dec 2011 02:56:42 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 362,880 597,174 1.6457
21 Dec 2011 16:09:46 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 336,960 558,001 1.6560
21 Dec 2011 03:36:40 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 311,040 515,228 1.6565
20 Dec 2011 15:00:49 1180747 13741205 hadcm3n_004y_1940_40_007607663_2 285,120 471,858 1.6549


©2024 cpdn.org