 |
|
| Name | hadcm3n_t0fo_1940_40_007446795_3 |
| Workunit | 7644298 |
| Created | 5 Oct 2011, 20:53:13 UTC |
| Sent | 5 Oct 2011, 20:53:27 UTC |
| Report deadline | 5 Jan 2012, 4:20:38 UTC |
| Received | 4 Jan 2012, 9:37:16 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 193 (0x000000C1) EXIT_SIGNAL |
| Computer ID | 1173509 |
| Run time | 61 days 17 hours 46 min 59 sec |
| CPU time | 59 days 17 hours 8 min 12 sec |
| Validate state | Invalid |
| Credit | 12,441.60 |
| Device peak FLOPS | 0.89 GFLOPS |
| Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu |
| Stderr | <core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:54:27 (1555): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09c36b90 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x6ebc2)[0xb51bc2] /lib/i386-linux-gnu/libc.so.6(+0x6f862)[0xb52862] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0xb5594d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0x8f180f] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1b)[0x8f186b] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xafc113] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 00110000-00113000 r-xp 00000000 08:01 523411 /lib/i386-linux-gnu/libdl-2.13.so 00113000-00114000 r--p 00002000 08:01 523411 /lib/i386-linux-gnu/libdl-2.13.so 00114000-00115000 rw-p 00003000 08:01 523411 /lib/i386-linux-gnu/libdl-2.13.so 00306000-0032e000 r-xp 00000000 08:01 523412 /lib/i386-linux-gnu/libm-2.13.so 0032e000-0032f000 r--p 00028000 08:01 523412 /lib/i386-linux-gnu/libm-2.13.so 0032f000-00330000 rw-p 00029000 08:01 523412 /lib/i386-linux-gnu/libm-2.13.so 00373000-0038f000 r-xp 00000000 08:01 523393 /lib/i386-linux-gnu/libgcc_s.so.1 0038f000-00390000 r--p 0001b000 08:01 523393 /lib/i386-linux-gnu/libgcc_s.so.1 00390000-00391000 rw-p 0001c000 08:01 523393 /lib/i386-linux-gnu/libgcc_s.so.1 00564000-00582000 r-xp 00000000 08:01 523395 /lib/i386-linux-gnu/ld-2.13.so 00582000-00583000 r--p 0001d000 08:01 523395 /lib/i386-linux-gnu/ld-2.13.so 00583000-00584000 rw-p 0001e000 08:01 523395 /lib/i386-linux-gnu/ld-2.13.so 00844000-00922000 r-xp 00000000 08:01 657831 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 00922000-00923000 ---p 000de000 08:01 657831 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 00923000-00927000 r--p 000de000 08:01 657831 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 00927000-00928000 rw-p 000e2000 08:01 657831 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 00928000-0092f000 rw-p 00000000 00:00 0 0097d000-00994000 r-xp 00000000 08:01 523504 /lib/i386-linux-gnu/libpthread-2.13.so 00994000-00995000 r--p 00016000 08:01 523504 /lib/i386-linux-gnu/libpthread-2.13.so 00995000-00996000 rw-p 00017000 08:01 523504 /lib/i386-linux-gnu/libpthread-2.13.so 00996000-00998000 rw-p 00000000 00:00 0 009e7000-009e8000 r-xp 00000000 00:00 0 [vdso] 00ae3000-00c59000 r-xp 00000000 08:01 523402 /lib/i386-linux-gnu/libc-2.13.so 00c59000-00c5b000 r--p 00176000 08:01 523402 /lib/i386-linux-gnu/libc-2.13.so 00c5b000-00c5c000 rw-p 00178000 08:01 523402 /lib/i386-linux-gnu/libc-2.13.so 00c5c000-00c5f000 rw-p 00000000 00:00 0 08048000-080e3000 r-xp 00000000 08:01 1575964 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 1575964 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 09be1000-09c46000 rw-p 00000000 00:00 0 [heap] b7100000-b7121000 rw-p 00000000 00:00 0 b7121000-b7200000 ---p 00000000 00:00 0 b72b6000-b7731000 rw-s 00000000 08:01 1705568 /var/lib/boinc-client/slots/0/137980 b7731000-b7735000 rw-p 00000000 00:00 0 b773f000-b7740000 rw-p 00000000 00:00 0 b7740000-b7741000 ---p 00000000 00:00 0 b7741000-b7744000 rw-p 00000000 00:00 0 b7744000-b7746000 rw-s 00000000 08:01 1705138 /var/lib/boinc-client/slots/0/boinc_mmap_file b7746000-b7748000 rw-p 00000000 00:00 0 bfe5d000-bfecc000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x9e7400] [0x9e7416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb10c8f] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb142b5] /lib/i386-linux-gnu/libc.so.6(+0x63dfc)[0xb46dfc] /lib/i386-linux-gnu/libc.so.6(+0x6ebc2)[0xb51bc2] /lib/i386-linux-gnu/libc.so.6(+0x6f862)[0xb52862] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0xb5594d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0x8f180f] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1b)[0x8f186b] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xafc113] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]> |
| Latest Trickles Received | ||||||
|---|---|---|---|---|---|---|
| Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
| 04 Jan 2012 08:41:10 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 1,036,800 | 5,159,284 | 4.9762 |
| 01 Jan 2012 17:14:22 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 1,010,880 | 5,029,482 | 4.9754 |
| 29 Dec 2011 16:26:15 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 984,960 | 4,901,599 | 4.9764 |
| 26 Dec 2011 13:05:59 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 959,040 | 4,773,486 | 4.9774 |
| 23 Dec 2011 19:10:09 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 933,120 | 4,645,605 | 4.9786 |
| 22 Dec 2011 06:47:53 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 907,200 | 4,517,470 | 4.9796 |
| 20 Dec 2011 18:49:46 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 881,280 | 4,388,888 | 4.9801 |
| 19 Dec 2011 05:31:55 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 855,360 | 4,258,914 | 4.9791 |
| 17 Dec 2011 17:15:54 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 829,440 | 4,130,602 | 4.9800 |
| 16 Dec 2011 04:50:31 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 803,520 | 4,002,026 | 4.9806 |
| 14 Dec 2011 16:30:16 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 777,600 | 3,873,477 | 4.9813 |
| 13 Dec 2011 03:59:17 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 751,680 | 3,744,585 | 4.9816 |
| 11 Dec 2011 14:48:19 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 725,760 | 3,613,224 | 4.9785 |
| 10 Dec 2011 01:09:13 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 699,840 | 3,480,091 | 4.9727 |
| 08 Dec 2011 10:13:36 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 673,920 | 3,353,032 | 4.9754 |
| 06 Dec 2011 21:51:36 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 648,000 | 3,224,915 | 4.9767 |
| 05 Dec 2011 09:21:03 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 622,080 | 3,096,232 | 4.9772 |
| 03 Dec 2011 20:01:16 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 596,160 | 2,964,652 | 4.9729 |
| 02 Dec 2011 06:46:20 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 570,240 | 2,833,152 | 4.9684 |
| 30 Nov 2011 17:43:15 | 1173509 | 13463324 | hadcm3n_t0fo_1940_40_007446795_3 | 544,320 | 2,702,345 | 4.9646 |
©2024 cpdn.org