Task 14279894

Name	hadcm3n_o1zk_1980_40_007831380_3
Workunit	7986492
Created	17 Mar 2012, 20:22:11 UTC
Sent	17 Mar 2012, 20:23:52 UTC
Report deadline	17 Jun 2012, 3:51:03 UTC
Received	17 May 2012, 8:26:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1207479
Run time	32 days 20 hours 57 min 37 sec
CPU time	30 days 14 hours 23 min 32 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	1.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb7792400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb74f0ca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb77e3400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7541ca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb7830400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb758eca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb76e9400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7447ca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb77fe400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb755cca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb7724400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7482ca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb77e3400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7541ca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb781d400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb757bca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb779e400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb74fcca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb77ce400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb752cca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb788a400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb75e8ca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb77c1400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb751fca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb7898400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb75f6ca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb787e400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb75dcca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb786e400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb75ccca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb78dc400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb763aca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb7749400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb74a7ca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09f7f7c8 *** ======= Backtrace: ========= /lib/i686/cmov/libc.so.6(+0x6b381)[0xb758a381] /lib/i686/cmov/libc.so.6(+0x6cbd8)[0xb758bbd8] /lib/i686/cmov/libc.so.6(cfree+0x6d)[0xb758ecbd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0xb7765701] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0xb776575d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7535ca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 08:05 1097735 /home/andre/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rwxp 0009b000 08:05 1097735 /home/andre/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rwxp 00000000 00:00 0 09f2a000-09f8f000 rwxp 00000000 00:00 0 [heap] b6f00000-b6f21000 rwxp 00000000 00:00 0 b6f21000-b7000000 ---p 00000000 00:00 0 b70a2000-b751d000 rwxs 00000000 08:05 1098075 /home/andre/BOINC/slots/2/137205 b751d000-b751f000 rwxp 00000000 00:00 0 b751f000-b765f000 r-xp 00000000 08:05 90550 /lib/i686/cmov/libc-2.11.3.so b765f000-b7660000 ---p 00140000 08:05 90550 /lib/i686/cmov/libc-2.11.3.so b7660000-b7662000 r-xp 00140000 08:05 90550 /lib/i686/cmov/libc-2.11.3.so b7662000-b7663000 rwxp 00142000 08:05 90550 /lib/i686/cmov/libc-2.11.3.so b7663000-b7666000 rwxp 00000000 00:00 0 b7666000-b7683000 r-xp 00000000 08:05 81923 /lib/libgcc_s.so.1 b7683000-b7684000 rwxp 0001c000 08:05 81923 /lib/libgcc_s.so.1 b7684000-b76a8000 r-xp 00000000 08:05 90531 /lib/i686/cmov/libm-2.11.3.so b76a8000-b76a9000 r-xp 00023000 08:05 90531 /lib/i686/cmov/libm-2.11.3.so b76a9000-b76aa000 rwxp 00024000 08:05 90531 /lib/i686/cmov/libm-2.11.3.so b76aa000-b7793000 r-xp 00000000 08:05 411677 /usr/lib/libstdc++.so.6.0.13 b7793000-b7797000 r-xp 000e9000 08:05 411677 /usr/lib/libstdc++.so.6.0.13 b7797000-b7798000 rwxp 000ed000 08:05 411677 /usr/lib/libstdc++.so.6.0.13 b7798000-b77a0000 rwxp 00000000 00:00 0 b77a0000-b77a2000 r-xp 00000000 08:05 90529 /lib/i686/cmov/libdl-2.11.3.so b77a2000-b77a3000 r-xp 00001000 08:05 90529 /lib/i686/cmov/libdl-2.11.3.so b77a3000-b77a4000 rwxp 00002000 08:05 90529 /lib/i686/cmov/libdl-2.11.3.so b77a4000-b77b9000 r-xp 00000000 08:05 90539 /lib/i686/cmov/libpthread-2.11.3.so b77b9000-b77ba000 r-xp 00014000 08:05 90539 /lib/i686/cmov/libpthread-2.11.3.so b77ba000-b77bb000 rwxp 00015000 08:05 90539 /lib/i686/cmov/libpthread-2.11.3.so b77bb000-b77bd000 rwxp 00000000 00:00 0 b77ce000-b77cf000 rwxp 00000000 00:00 0 b77cf000-b77d0000 ---p 00000000 00:00 0 b77d0000-b77d3000 rwxp 00000000 00:00 0 b77d3000-b77d5000 rwxs 00000000 08:05 1097952 /home/andre/BOINC/slots/2/boinc_mmap_file b77d5000-b77d7000 rwxp 00000000 00:00 0 b77d7000-b77d8000 r-xp 00000000 00:00 0 [vdso] b77d8000-b77f3000 r-xp 00000000 08:05 83307 /lib/ld-2.11.3.so b77f3000-b77f4000 r-xp 0001b000 08:05 83307 /lib/ld-2.11.3.so b77f4000-b77f5000 rwxp 0001c000 08:05 83307 /lib/ld-2.11.3.so bfbb1000-bfc20000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb77d7400] [0xb77d7424] /lib/i686/cmov/libc.so.6(gsignal+0x51)[0xb7549781] /lib/i686/cmov/libc.so.6(abort+0x182)[0xb754cbb2] /lib/i686/cmov/libc.so.6(+0x61285)[0xb7580285] /lib/i686/cmov/libc.so.6(+0x6b381)[0xb758a381] /lib/i686/cmov/libc.so.6(+0x6cbd8)[0xb758bbd8] /lib/i686/cmov/libc.so.6(cfree+0x6d)[0xb758ecbd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0xb7765701] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0xb776575d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7535ca6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 May 2012 02:20:39	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	1,036,800	2,643,805	2.5500
13 May 2012 17:27:48	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	1,010,880	2,573,563	2.5459
12 May 2012 20:16:44	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	984,960	2,502,837	2.5411
12 May 2012 00:51:33	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	959,040	2,433,784	2.5377
11 May 2012 08:51:54	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	933,120	2,367,467	2.5372
06 May 2012 22:30:26	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	907,200	2,301,792	2.5372
06 May 2012 04:08:03	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	881,280	2,236,231	2.5375
05 May 2012 10:30:56	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	855,360	2,172,870	2.5403
01 May 2012 21:57:29	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	829,440	2,106,675	2.5399
01 May 2012 04:11:24	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	803,520	2,043,143	2.5427
30 Apr 2012 10:11:03	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	777,600	1,978,770	2.5447
29 Apr 2012 15:35:51	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	751,680	1,913,030	2.5450
28 Apr 2012 22:02:19	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	725,760	1,849,895	2.5489
28 Apr 2012 04:21:16	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	699,840	1,786,538	2.5528
26 Apr 2012 03:08:38	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	673,920	1,719,456	2.5514
25 Apr 2012 13:38:49	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	648,000	1,653,847	2.5522
22 Apr 2012 05:46:34	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	622,080	1,591,088	2.5577
21 Apr 2012 19:48:01	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	596,160	1,528,310	2.5636
18 Apr 2012 03:41:37	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	570,240	1,465,285	2.5696
17 Apr 2012 00:24:18	1207479	14279894	hadcm3n_o1zk_1980_40_007831380_3	544,320	1,401,738	2.5752