Task 15473627

Name	hadcm3n_zlsu_1880_40_008246959_2
Workunit	8402083
Created	7 Dec 2012, 18:54:37 UTC
Sent	7 Dec 2012, 18:54:43 UTC
Report deadline	9 Mar 2013, 2:21:54 UTC
Received	30 Jan 2013, 19:49:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1142456
Run time	18 days 8 hours 20 min 7 sec
CPU time	16 days 22 hours 21 min 58 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	0.93 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.59</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:31:44 (26476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:31:45 (26476): No heartbeat from core client for 30 sec - exiting 00:31:46 (26476): No heartbeat from core client for 30 sec - exiting 00:31:47 (26476): No heartbeat from core client for 30 sec - exiting 00:31:48 (26476): No heartbeat from core client for 30 sec - exiting 00:31:49 (26476): No heartbeat from core client for 30 sec - exiting 00:31:50 (26476): No heartbeat from core client for 30 sec - exiting 00:31:51 (26476): No heartbeat from core client for 30 sec - exiting 00:31:52 (26476): No heartbeat from core client for 30 sec - exiting 00:31:53 (26476): No heartbeat from core client for 30 sec - exiting 00:31:54 (26476): No heartbeat from core client for 30 sec - exiting 00:31:55 (26476): No heartbeat from core client for 30 sec - exiting 00:31:56 (26476): No heartbeat from core client for 30 sec - exiting 00:31:57 (26476): No heartbeat from core client for 30 sec - exiting 00:31:58 (26476): No heartbeat from core client for 30 sec - exiting 00:31:59 (26476): No heartbeat from core client for 30 sec - exiting 00:32:00 (26476): No heartbeat from core client for 30 sec - exiting 00:32:01 (26476): No heartbeat from core client for 30 sec - exiting 00:32:02 (26476): No heartbeat from core client for 30 sec - exiting 00:32:03 (26476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:49:37 (2998): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08aeacc8 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x6f751)[0x563751] /lib/i386-linux-gnu/libc.so.6(+0x7107b)[0x56507b] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0x56820d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x21)[0x21e4d1] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1d)[0x21e52d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x50ae37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 00174000-00253000 r-xp 00000000 08:01 6299 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 00253000-00257000 r--p 000de000 08:01 6299 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 00257000-00258000 rw-p 000e2000 08:01 6299 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 00258000-0025f000 rw-p 00000000 00:00 0 003ad000-003c7000 r-xp 00000000 08:01 1043008 /lib/i386-linux-gnu/libgcc_s.so.1 003c7000-003c8000 r--p 00019000 08:01 1043008 /lib/i386-linux-gnu/libgcc_s.so.1 003c8000-003c9000 rw-p 0001a000 08:01 1043008 /lib/i386-linux-gnu/libgcc_s.so.1 0040b000-0040c000 r-xp 00000000 00:00 0 [vdso] 004d6000-004f2000 r-xp 00000000 08:01 1059050 /lib/i386-linux-gnu/ld-2.13.so 004f2000-004f3000 r--p 0001b000 08:01 1059050 /lib/i386-linux-gnu/ld-2.13.so 004f3000-004f4000 rw-p 0001c000 08:01 1059050 /lib/i386-linux-gnu/ld-2.13.so 004f4000-00655000 r-xp 00000000 08:01 1059950 /lib/i386-linux-gnu/libc-2.13.so 00655000-00656000 ---p 00161000 08:01 1059950 /lib/i386-linux-gnu/libc-2.13.so 00656000-00658000 r--p 00161000 08:01 1059950 /lib/i386-linux-gnu/libc-2.13.so 00658000-00659000 rw-p 00163000 08:01 1059950 /lib/i386-linux-gnu/libc-2.13.so 00659000-0065c000 rw-p 00000000 00:00 0 0084b000-00860000 r-xp 00000000 08:01 1059951 /lib/i386-linux-gnu/libpthread-2.13.so 00860000-00861000 r--p 00015000 08:01 1059951 /lib/i386-linux-gnu/libpthread-2.13.so 00861000-00862000 rw-p 00016000 08:01 1059951 /lib/i386-linux-gnu/libpthread-2.13.so 00862000-00864000 rw-p 00000000 00:00 0 00b92000-00b94000 r-xp 00000000 08:01 1059944 /lib/i386-linux-gnu/libdl-2.13.so 00b94000-00b95000 r--p 00001000 08:01 1059944 /lib/i386-linux-gnu/libdl-2.13.so 00b95000-00b96000 rw-p 00002000 08:01 1059944 /lib/i386-linux-gnu/libdl-2.13.so 00d31000-00d55000 r-xp 00000000 08:01 1059952 /lib/i386-linux-gnu/libm-2.13.so 00d55000-00d56000 r--p 00023000 08:01 1059952 /lib/i386-linux-gnu/libm-2.13.so 00d56000-00d57000 rw-p 00024000 08:01 1059952 /lib/i386-linux-gnu/libm-2.13.so 08048000-080e3000 r-xp 00000000 08:01 1583169 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 1583169 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 08a95000-08afa000 rw-p 00000000 00:00 0 [heap] b7100000-b7121000 rw-p 00000000 00:00 0 b7121000-b7200000 ---p 00000000 00:00 0 b72d0000-b774b000 rw-s 00000000 08:01 1321613 /var/lib/boinc-client/slots/2/142295 b774b000-b774f000 rw-p 00000000 00:00 0 b7766000-b7767000 rw-p 00000000 00:00 0 b7767000-b7768000 ---p 00000000 00:00 0 b7768000-b776b000 rw-p 00000000 00:00 0 b776b000-b776d000 rw-s 00000000 08:01 1342390 /var/lib/boinc-client/slots/2/boinc_mmap_file b776d000-b776f000 rw-p 00000000 00:00 0 bfae1000-bfb50000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x40b400] [0x40b416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x51)[0x51ee71] /lib/i386-linux-gnu/libc.so.6(abort+0x17e)[0x52234e] /lib/i386-linux-gnu/libc.so.6(+0x65367)[0x559367] /lib/i386-linux-gnu/libc.so.6(+0x6f751)[0x563751] /lib/i386-linux-gnu/libc.so.6(+0x7107b)[0x56507b] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0x56820d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x21)[0x21e4d1] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1d)[0x21e52d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x50ae37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Jan 2013 19:51:44	1142456	15473627	hadcm3n_zlsu_1880_40_008246959_2	259,200	1,462,907	5.6439
27 Jan 2013 19:18:21	1142456	15473627	hadcm3n_zlsu_1880_40_008246959_2	233,280	1,315,293	5.6383
24 Jan 2013 12:35:08	1142456	15473627	hadcm3n_zlsu_1880_40_008246959_2	207,360	1,167,154	5.6286
21 Jan 2013 18:51:49	1142456	15473627	hadcm3n_zlsu_1880_40_008246959_2	181,440	1,020,409	5.6239
17 Jan 2013 13:50:55	1142456	15473627	hadcm3n_zlsu_1880_40_008246959_2	155,520	873,376	5.6158
30 Dec 2012 22:16:01	1142456	15473627	hadcm3n_zlsu_1880_40_008246959_2	129,600	729,829	5.6314
27 Dec 2012 11:50:00	1142456	15473627	hadcm3n_zlsu_1880_40_008246959_2	103,680	580,869	5.6025
19 Dec 2012 18:29:56	1142456	15473627	hadcm3n_zlsu_1880_40_008246959_2	77,760	435,786	5.6042
15 Dec 2012 02:12:17	1142456	15473627	hadcm3n_zlsu_1880_40_008246959_2	51,840	292,010	5.6329
13 Dec 2012 22:32:38	1142456	15473627	hadcm3n_zlsu_1880_40_008246959_2	25,920	146,370	5.6470