Task 17357928

Name	hadcm3n_xd3u_1940_40_009153280_0
Workunit	9283616
Created	6 Nov 2014, 15:54:35 UTC
Sent	6 Nov 2014, 18:15:04 UTC
Report deadline	6 Feb 2015, 1:42:15 UTC
Received	27 Dec 2014, 21:30:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1341772
Run time	18 days 23 hours 53 min 3 sec
CPU time	18 days 19 hours 46 min 46 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	1.83 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:32:19 (5416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09e74300 * ======= Backtrace: ========= /lib/libc.so.6[0x452afad9] /lib/libc.so.6[0x452b70d6] /lib/libc.so.6(cfree+0x56)[0x452bb346] /lib/libstdc++.so.6(_ZdlPv+0x1c)[0x454e101c] /lib/libstdc++.so.6(_ZdaPv+0x1c)[0x454e107c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xde)[0x45259e7e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 08:02 1706547 /var/lib/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:02 1706547 /var/lib/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 09e1e000-09ea5000 rw-p 00000000 00:00 0 [heap] 4521c000-4523e000 r-xp 00000000 08:02 269290 /usr/lib/ld-2.20.so 4523e000-4523f000 r--p 00021000 08:02 269290 /usr/lib/ld-2.20.so 4523f000-45240000 rw-p 00022000 08:02 269290 /usr/lib/ld-2.20.so 45242000-45408000 r-xp 00000000 08:02 269292 /usr/lib/libc-2.20.so 45408000-4540b000 r--p 001c5000 08:02 269292 /usr/lib/libc-2.20.so 4540b000-4540d000 rw-p 001c8000 08:02 269292 /usr/lib/libc-2.20.so 4540d000-4540f000 rw-p 00000000 00:00 0 45411000-45428000 r-xp 00000000 08:02 271167 /usr/lib/libpthread-2.20.so 45428000-45429000 r--p 00016000 08:02 271167 /usr/lib/libpthread-2.20.so 45429000-4542a000 rw-p 00017000 08:02 271167 /usr/lib/libpthread-2.20.so 4542a000-4542c000 rw-p 00000000 00:00 0 4542e000-45431000 r-xp 00000000 08:02 271484 /usr/lib/libdl-2.20.so 45431000-45432000 r--p 00002000 08:02 271484 /usr/lib/libdl-2.20.so 45432000-45433000 rw-p 00003000 08:02 271484 /usr/lib/libdl-2.20.so 45440000-4548b000 r-xp 00000000 08:02 271193 /usr/lib/libm-2.20.so 4548b000-4548c000 r--p 0004b000 08:02 271193 /usr/lib/libm-2.20.so 4548c000-4548d000 rw-p 0004c000 08:02 271193 /usr/lib/libm-2.20.so 45495000-45583000 r-xp 00000000 08:02 272085 /usr/lib/libstdc++.so.6.0.20 45583000-45587000 r--p 000ee000 08:02 272085 /usr/lib/libstdc++.so.6.0.20 45587000-45588000 rw-p 000f2000 08:02 272085 /usr/lib/libstdc++.so.6.0.20 45588000-4558f000 rw-p 00000000 00:00 0 45768000-45784000 r-xp 00000000 08:02 269443 /usr/lib/libgcc_s-4.9.2-20141101.so.1 45784000-45785000 r--p 0001b000 08:02 269443 /usr/lib/libgcc_s-4.9.2-20141101.so.1 45785000-45786000 rw-p 0001c000 08:02 269443 /usr/lib/libgcc_s-4.9.2-20141101.so.1 f7305000-f7780000 rw-s 00000000 08:02 1836071 /var/lib/boinc/slots/1/136325 f7780000-f7784000 rw-p 00000000 00:00 0 f77a4000-f77a6000 rw-p 00000000 00:00 0 f77a6000-f77a7000 ---p 00000000 00:00 0 f77a7000-f77aa000 rw-p 00000000 00:00 0 [stack:1104] f77aa000-f77ac000 rw-s 00000000 08:02 1836068 /var/lib/boinc/slots/1/boinc_mmap_file f77ac000-f77ad000 rw-p 00000000 00:00 0 f77ad000-f77af000 r--p 00000000 00:00 0 [vvar] f77af000-f77b0000 r-xp 00000000 00:00 0 [vdso] ffee5000-fff56000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (18 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf77afbc0] [0xf77afbe5] /lib/libc.so.6(gsignal+0x47)[0x4526f297] /lib/libc.so.6(abort+0x149)[0x45270b69] /lib/libc.so.6[0x452afade] /lib/libc.so.6[0x452b70d6] /lib/libc.so.6(cfree+0x56)[0x452bb346] /lib/libstdc++.so.6(_ZdlPv+0x1c)[0x454e101c] /lib/libstdc++.so.6(_ZdaPv+0x1c)[0x454e107c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xde)[0x45259e7e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Dec 2014 20:30:40	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	1,036,800	1,626,398	1.5687
26 Dec 2014 21:40:18	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	1,010,880	1,585,312	1.5682
25 Dec 2014 20:04:54	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	984,960	1,545,430	1.5690
23 Dec 2014 20:10:44	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	959,040	1,505,197	1.5695
22 Dec 2014 19:24:13	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	933,120	1,465,341	1.5704
20 Dec 2014 21:35:36	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	907,200	1,425,527	1.5713
19 Dec 2014 11:16:10	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	881,280	1,386,155	1.5729
17 Dec 2014 20:03:54	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	855,360	1,346,279	1.5739
17 Dec 2014 08:16:50	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	829,440	1,304,975	1.5733
15 Dec 2014 13:10:20	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	803,520	1,262,620	1.5714
13 Dec 2014 19:14:41	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	777,600	1,221,924	1.5714
12 Dec 2014 10:18:26	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	751,680	1,181,718	1.5721
11 Dec 2014 12:19:16	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	725,760	1,139,400	1.5699
10 Dec 2014 09:47:32	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	699,840	1,098,701	1.5699
08 Dec 2014 08:41:53	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	673,920	1,057,712	1.5695
07 Dec 2014 21:01:57	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	648,000	1,015,950	1.5678
07 Dec 2014 06:30:17	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	622,080	974,911	1.5672
06 Dec 2014 19:18:23	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	596,160	934,679	1.5678
05 Dec 2014 14:11:06	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	570,240	893,101	1.5662
03 Dec 2014 18:58:54	1341772	17357928	hadcm3n_xd3u_1940_40_009153280_0	544,320	852,184	1.5656