 |
|
| Name | hadcm3n_o6ua_1980_40_007752421_2 |
| Workunit | 7907530 |
| Created | 19 Feb 2012, 10:04:41 UTC |
| Sent | 19 Feb 2012, 10:04:44 UTC |
| Report deadline | 20 May 2012, 17:31:55 UTC |
| Received | 30 Mar 2012, 10:08:34 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 193 (0x000000C1) EXIT_SIGNAL |
| Computer ID | 1189274 |
| Run time | 20 days 14 hours 12 min 54 sec |
| CPU time | 19 days 23 hours 50 min 37 sec |
| Validate state | Invalid |
| Credit | 6,220.80 |
| Device peak FLOPS | 1.56 GFLOPS |
| Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu |
| Stderr | <core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=18960, selfPID=18960, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/o6uako.pjj9c10 is not a valid UM file. Error converting file to netcdf: dataout/o6uako.pjj9c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08ea4850 *** ======= Backtrace: ========= /lib/libc.so.6(+0x6c0c1)[0x17c0c1] /lib/libc.so.6(+0x6d930)[0x17d930] /lib/libc.so.6(cfree+0x6d)[0x180a1d] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x392441] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x39249d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xe7)[0x126ce7] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 00110000-00267000 r-xp 00000000 08:01 3150532 /lib/libc-2.12.1.so 00267000-00269000 r--p 00157000 08:01 3150532 /lib/libc-2.12.1.so 00269000-0026a000 rw-p 00159000 08:01 3150532 /lib/libc-2.12.1.so 0026a000-0026d000 rw-p 00000000 00:00 0 002e8000-003c7000 r-xp 00000000 08:01 1707605 /usr/lib/libstdc++.so.6.0.14 003c7000-003cb000 r--p 000de000 08:01 1707605 /usr/lib/libstdc++.so.6.0.14 003cb000-003cc000 rw-p 000e2000 08:01 1707605 /usr/lib/libstdc++.so.6.0.14 003cc000-003d3000 rw-p 00000000 00:00 0 0073a000-0075e000 r-xp 00000000 08:01 3150490 /lib/libm-2.12.1.so 0075e000-0075f000 r--p 00023000 08:01 3150490 /lib/libm-2.12.1.so 0075f000-00760000 rw-p 00024000 08:01 3150490 /lib/libm-2.12.1.so 00791000-007ab000 r-xp 00000000 08:01 3145807 /lib/libgcc_s.so.1 007ab000-007ac000 r--p 00019000 08:01 3145807 /lib/libgcc_s.so.1 007ac000-007ad000 rw-p 0001a000 08:01 3145807 /lib/libgcc_s.so.1 00924000-00925000 r-xp 00000000 00:00 0 [vdso] 009d4000-009d6000 r-xp 00000000 08:01 3150475 /lib/libdl-2.12.1.so 009d6000-009d7000 r--p 00001000 08:01 3150475 /lib/libdl-2.12.1.so 009d7000-009d8000 rw-p 00002000 08:01 3150475 /lib/libdl-2.12.1.so 00d08000-00d24000 r-xp 00000000 08:01 3147364 /lib/ld-2.12.1.so 00d24000-00d25000 r--p 0001b000 08:01 3147364 /lib/ld-2.12.1.so 00d25000-00d26000 rw-p 0001c000 08:01 3147364 /lib/ld-2.12.1.so 00e3b000-00e50000 r-xp 00000000 08:01 3150535 /lib/libpthread-2.12.1.so 00e50000-00e51000 ---p 00015000 08:01 3150535 /lib/libpthread-2.12.1.so 00e51000-00e52000 r--p 00015000 08:01 3150535 /lib/libpthread-2.12.1.so 00e52000-00e53000 rw-p 00016000 08:01 3150535 /lib/libpthread-2.12.1.so 00e53000-00e55000 rw-p 00000000 00:00 0 08048000-080e3000 r-xp 00000000 08:01 4589283 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 4589283 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 08e4f000-08eb4000 rw-p 00000000 00:00 0 [heap] b7200000-b7221000 rw-p 00000000 00:00 0 b7221000-b7300000 ---p 00000000 00:00 0 b73c7000-b7842000 rw-s 00000000 08:01 4718714 /var/lib/boinc-client/slots/3/136330 b7842000-b7846000 rw-p 00000000 00:00 0 b7853000-b7854000 rw-p 00000000 00:00 0 b7854000-b7855000 ---p 00000000 00:00 0 b7855000-b7858000 rw-p 00000000 00:00 0 b7858000-b785a000 rw-s 00000000 08:01 4718624 /var/lib/boinc-client/slots/3/boinc_mmap_file b785a000-b785c000 rw-p 00000000 00:00 0 bfae6000-bfb56000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x924400] [0x924416] /lib/libc.so.6(gsignal+0x51)[0x13a941] /lib/libc.so.6(abort+0x182)[0x13de42] /lib/libc.so.6(+0x61ec5)[0x171ec5] /lib/libc.so.6(+0x6c0c1)[0x17c0c1] /lib/libc.so.6(+0x6d930)[0x17d930] /lib/libc.so.6(cfree+0x6d)[0x180a1d] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x392441] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x39249d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xe7)[0x126ce7] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]> |
| Latest Trickles Received | ||||||
|---|---|---|---|---|---|---|
| Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
| 30 Mar 2012 06:59:22 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 518,400 | 1,727,426 | 3.3322 |
| 26 Mar 2012 18:22:34 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 492,480 | 1,640,729 | 3.3316 |
| 24 Mar 2012 13:21:06 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 466,560 | 1,556,250 | 3.3356 |
| 22 Mar 2012 02:24:17 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 440,640 | 1,465,154 | 3.3251 |
| 19 Mar 2012 16:25:42 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 414,720 | 1,374,315 | 3.3138 |
| 17 Mar 2012 20:17:24 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 388,800 | 1,287,700 | 3.3120 |
| 15 Mar 2012 17:22:30 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 362,880 | 1,195,529 | 3.2946 |
| 13 Mar 2012 19:52:39 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 336,960 | 1,109,430 | 3.2925 |
| 12 Mar 2012 19:23:18 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 311,040 | 1,025,522 | 3.2971 |
| 08 Mar 2012 21:50:08 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 285,120 | 942,573 | 3.3059 |
| 07 Mar 2012 20:55:20 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 259,200 | 859,514 | 3.3160 |
| 06 Mar 2012 20:35:01 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 233,280 | 778,844 | 3.3387 |
| 03 Mar 2012 12:44:01 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 207,360 | 697,281 | 3.3627 |
| 01 Mar 2012 21:13:32 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 181,440 | 607,373 | 3.3475 |
| 29 Feb 2012 18:42:16 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 155,520 | 515,340 | 3.3137 |
| 28 Feb 2012 14:14:16 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 129,600 | 419,189 | 3.2345 |
| 27 Feb 2012 12:15:27 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 103,680 | 329,113 | 3.1743 |
| 23 Feb 2012 21:29:59 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 77,760 | 243,246 | 3.1282 |
| 22 Feb 2012 12:42:49 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 51,840 | 165,715 | 3.1967 |
| 20 Feb 2012 10:00:39 | 1189274 | 14112007 | hadcm3n_o6ua_1980_40_007752421_2 | 25,920 | 84,670 | 3.2666 |
©2024 cpdn.org