Task 16039672

Name	hadcm3n_oaph_1900_40_008469000_0
Workunit	8619839
Created	27 Sep 2013, 9:44:09 UTC
Sent	4 Oct 2013, 11:16:15 UTC
Report deadline	3 Jan 2014, 18:43:26 UTC
Received	23 Nov 2013, 20:27:44 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	255 (0x000000FF) Unknown error code
Computer ID	895306
Run time	32 days 10 hours 19 min 8 sec
CPU time	19 days 0 hours 46 min 36 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.49 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.28</core_client_version> <![CDATA[ <message> The extended attributes are inconsistent. (0xff) - exit code 255 (0xff) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:17:03 (308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:17:04 (308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:57:24 (2264): No heartbeat from core client for 30 sec - exiting 18:57:25 (2264): No heartbeat from core client for 30 sec - exiting 18:57:26 (2264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Ocean Restart file copy failed on oaphko.daa4af0 CPDN Monitor - Quit request from BOINC... 22:45:03 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:33:02 (5668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:01:36 (6052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:09:15 (5648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:53:31 (5740): No heartbeat from core client for 30 sec - exiting 22:53:32 (5740): No heartbeat from core client for 30 sec - exiting 22:53:33 (5740): No heartbeat from core client for 30 sec - exiting 22:53:34 (5740): No heartbeat from core client for 30 sec - exiting 22:53:35 (5740): No heartbeat from core client for 30 sec - exiting 22:53:36 (5740): No heartbeat from core client for 30 sec - exiting 22:53:38 (5740): No heartbeat from core client for 30 sec - exiting 22:53:39 (5740): No heartbeat from core client for 30 sec - exiting 22:53:40 (5740): No heartbeat from core client for 30 sec - exiting 22:53:41 (5740): No heartbeat from core client for 30 sec - exiting 22:53:42 (5740): No heartbeat from core client for 30 sec - exiting 22:53:43 (5740): No heartbeat from core client for 30 sec - exiting 22:53:44 (5740): No heartbeat from core client for 30 sec - exiting 22:53:45 (5740): No heartbeat from core client for 30 sec - exiting 22:53:46 (5740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:53:48 (5740): No heartbeat from core client for 30 sec - exiting 22:53:49 (5740): No heartbeat from core client for 30 sec - exiting 22:53:50 (5740): No heartbeat from core client for 30 sec - exiting 22:53:51 (5740): No heartbeat from core client for 30 sec - exiting 22:53:52 (5740): No heartbeat from core client for 30 sec - exiting 22:53:53 (5740): No heartbeat from core client for 30 sec - exiting 23:06:27 (5240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:06:29 (5240): No heartbeat from core client for 30 sec - exiting 23:06:30 (5240): No heartbeat from core client for 30 sec - exiting 23:42:09 (2916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:25:25 (5912): No heartbeat from core client for 30 sec - exiting 23:25:26 (5912): No heartbeat from core client for 30 sec - exiting 23:25:27 (5912): No heartbeat from core client for 30 sec - exiting 23:25:28 (5912): No heartbeat from core client for 30 sec - exiting 23:25:29 (5912): No heartbeat from core client for 30 sec - exiting 23:25:30 (5912): No heartbeat from core client for 30 sec - exiting 23:25:31 (5912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3856, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:10:39 (3180): No heartbeat from core client for 30 sec - exiting 13:10:40 (3180): No heartbeat from core client for 30 sec - exiting 13:10:41 (3180): No heartbeat from core client for 30 sec - exiting 13:10:42 (3180): No heartbeat from core client for 30 sec - exiting 13:10:43 (3180): No heartbeat from core client for 30 sec - exiting 13:10:44 (3180): No heartbeat from core client for 30 sec - exiting 13:10:45 (3180): No heartbeat from core client for 30 sec - exiting 13:10:46 (3180): No heartbeat from core client for 30 sec - exiting 13:10:47 (3180): No heartbeat from core client for 30 sec - exiting 13:10:48 (3180): No heartbeat from core client for 30 sec - exiting 13:10:49 (3180): No heartbeat from core client for 30 sec - exiting 13:10:50 (3180): No heartbeat from core client for 30 sec - exiting 13:10:51 (3180): No heartbeat from core client for 30 sec - exiting 13:10:52 (3180): No heartbeat from core client for 30 sec - exiting 13:10:53 (3180): No heartbeat from core client for 30 sec - exiting 13:10:54 (3180): No heartbeat from core client for 30 sec - exiting 13:10:55 (3180): No heartbeat from core client for 30 sec - exiting 13:10:57 (3180): No heartbeat from core client for 30 sec - exiting 13:10:58 (3180): No heartbeat from core client for 30 sec - exiting 13:10:59 (3180): No heartbeat from core client for 30 sec - exiting 13:11:00 (3180): No heartbeat from core client for 30 sec - exiting 13:11:01 (3180): No heartbeat from core client for 30 sec - exiting 13:11:02 (3180): No heartbeat from core client for 30 sec - exiting 13:11:03 (3180): No heartbeat from core client for 30 sec - exiting 13:11:04 (3180): No heartbeat from core client for 30 sec - exiting 13:11:05 (3180): No heartbeat from core client for 30 sec - exiting 13:11:06 (3180): No heartbeat from core client for 30 sec - exiting 13:11:07 (3180): No heartbeat from core client for 30 sec - exiting 13:11:08 (3180): No heartbeat from core client for 30 sec - exiting 13:11:09 (3180): No heartbeat from core client for 30 sec - exiting 13:11:10 (3180): No heartbeat from core client for 30 sec - exiting 13:11:11 (3180): No heartbeat from core client for 30 sec - exiting 13:11:13 (3180): No heartbeat from core client for 30 sec - exiting 13:11:14 (3180): No heartbeat from core client for 30 sec - exiting 13:11:16 (3180): No heartbeat from core client for 30 sec - exiting 13:11:17 (3180): No heartbeat from core client for 30 sec - exiting 13:11:18 (3180): No heartbeat from core client for 30 sec - exiting 13:11:19 (3180): No heartbeat from core client for 30 sec - exiting 13:11:20 (3180): No heartbeat from core client for 30 sec - exiting 13:11:21 (3180): No heartbeat from core client for 30 sec - exiting 13:11:22 (3180): No heartbeat from core client for 30 sec - exiting 13:11:23 (3180): No heartbeat from core client for 30 sec - exiting 13:11:24 (3180): No heartbeat from core client for 30 sec - exiting 13:11:25 (3180): No heartbeat from core client for 30 sec - exiting 13:11:26 (3180): No heartbeat from core client for 30 sec - exiting 13:11:27 (3180): No heartbeat from core client for 30 sec - exiting 13:11:29 (3180): No heartbeat from core client for 30 sec - exiting 13:11:30 (3180): No heartbeat from core client for 30 sec - exiting 13:11:31 (3180): No heartbeat from core client for 30 sec - exiting 13:11:32 (3180): No heartbeat from core client for 30 sec - exiting 13:11:33 (3180): No heartbeat from core client for 30 sec - exiting 13:11:34 (3180): No heartbeat from core client for 30 sec - exiting 13:11:35 (3180): No heartbeat from core client for 30 sec - exiting 13:11:36 (3180): No heartbeat from core client for 30 sec - exiting 13:11:37 (3180): No heartbeat from core client for 30 sec - exiting 13:11:38 (3180): No heartbeat from core client for 30 sec - exiting 13:11:39 (3180): No heartbeat from core client for 30 sec - exiting 13:11:40 (3180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:33:36 (5684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:33:37 (5684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:36:43 (7028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:44 (7028): No heartbeat from core client for 30 sec - exiting 11:36:45 (7028): No heartbeat from core client for 30 sec - exiting 11:36:46 (7028): No heartbeat from core client for 30 sec - exiting 19:01:21 (6512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:01:23 (6512): No heartbeat from core client for 30 sec - exiting 19:01:24 (6512): No heartbeat from core client for 30 sec - exiting 19:01:25 (6512): No heartbeat from core client for 30 sec - exiting 19:01:26 (6512): No heartbeat from core client for 30 sec - exiting 19:01:27 (6512): No heartbeat from core client for 30 sec - exiting 19:04:54 (9852): No heartbeat from core client for 30 sec - exiting 19:04:56 (9852): No heartbeat from core client for 30 sec - exiting 19:04:57 (9852): No heartbeat from core client for 30 sec - exiting 19:04:58 (9852): No heartbeat from core client for 30 sec - exiting 19:05:00 (9852): No heartbeat from core client for 30 sec - exiting 19:05:01 (9852): No heartbeat from core client for 30 sec - exiting 19:05:02 (9852): No heartbeat from core client for 30 sec - exiting 19:05:03 (9852): No heartbeat from core client for 30 sec - exiting 19:05:04 (9852): No heartbeat from core client for 30 sec - exiting 19:05:05 (9852): No heartbeat from core client for 30 sec - exiting 19:05:07 (9852): No heartbeat from core client for 30 sec - exiting 19:05:08 (9852): No heartbeat from core client for 30 sec - exiting 19:05:09 (9852): No heartbeat from core client for 30 sec - exiting 19:05:11 (9852): No heartbeat from core client for 30 sec - exiting 19:05:12 (9852): No heartbeat from core client for 30 sec - exiting 19:05:13 (9852): No heartbeat from core client for 30 sec - exiting 19:05:14 (9852): No heartbeat from core client for 30 sec - exiting 19:05:15 (9852): No heartbeat from core client for 30 sec - exiting 19:05:16 (9852): No heartbeat from core client for 30 sec - exiting 19:05:17 (9852): No heartbeat from core client for 30 sec - exiting 19:05:19 (9852): No heartbeat from core client for 30 sec - exiting 19:05:28 (9852): No heartbeat from core client for 30 sec - exiting 19:05:29 (9852): No heartbeat from core client for 30 sec - exiting 19:05:31 (9852): No heartbeat from core client for 30 sec - exiting 19:05:32 (9852): No heartbeat from core client for 30 sec - exiting 19:05:33 (9852): No heartbeat from core client for 30 sec - exiting 19:05:34 (9852): No heartbeat from core client for 30 sec - exiting 19:05:43 (9852): No heartbeat from core client for 30 sec - exiting 19:05:44 (9852): No heartbeat from core client for 30 sec - exiting 19:05:45 (9852): No heartbeat from core client for 30 sec - exiting 19:05:46 (9852): No heartbeat from core client for 30 sec - exiting 19:05:48 (9852): No heartbeat from core client for 30 sec - exiting 19:05:49 (9852): No heartbeat from core client for 30 sec - exiting 19:05:50 (9852): No heartbeat from core client for 30 sec - exiting 19:05:51 (9852): No heartbeat from core client for 30 sec - exiting 19:05:53 (9852): No heartbeat from core client for 30 sec - exiting 19:05:54 (9852): No heartbeat from core client for 30 sec - exiting 19:05:55 (9852): No heartbeat from core client for 30 sec - exiting 19:05:56 (9852): No heartbeat from core client for 30 sec - exiting 19:05:57 (9852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:43:11 (8572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:43:13 (8572): No heartbeat from core client for 30 sec - exiting 19:43:14 (8572): No heartbeat from core client for 30 sec - exiting 19:43:15 (8572): No heartbeat from core client for 30 sec - exiting 19:43:16 (8572): No heartbeat from core client for 30 sec - exiting 19:53:09 (5008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:53:15 (5008): No heartbeat from core client for 30 sec - exiting 19:53:16 (5008): No heartbeat from core client for 30 sec - exiting 19:53:18 (5008): No heartbeat from core client for 30 sec - exiting 19:53:19 (5008): No heartbeat from core client for 30 sec - exiting 19:53:20 (5008): No heartbeat from core client for 30 sec - exiting 20:13:17 (6668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:13:21 (6668): No heartbeat from core client for 30 sec - exiting 20:13:22 (6668): No heartbeat from core client for 30 sec - exiting 00:19:18 (1708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:22:36 (8680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:22:38 (8680): No heartbeat from core client for 30 sec - exiting 01:22:39 (8680): No heartbeat from core client for 30 sec - exiting 01:22:40 (8680): No heartbeat from core client for 30 sec - exiting 01:22:41 (8680): No heartbeat from core client for 30 sec - exiting 01:22:42 (8680): No heartbeat from core client for 30 sec - exiting 02:16:37 (6208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:08:55 (8464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:08:56 (8464): No heartbeat from core client for 30 sec - exiting 16:08:57 (8464): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9496, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:33:57 (4232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:40:19 (5964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:40:20 (5964): No heartbeat from core client for 30 sec - exiting 23:41:45 (11168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10932, iMonCtr=1 Model crash detected, will try to restart... 02:27:10 (6676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:27:11 (6676): No heartbeat from core client for 30 sec - exiting 02:27:13 (6676): No heartbeat from core client for 30 sec - exiting 02:27:14 (6676): No heartbeat from core client for 30 sec - exiting 02:27:15 (6676): No heartbeat from core client for 30 sec - exiting 02:27:16 (6676): No heartbeat from core client for 30 sec - exiting 02:27:17 (6676): No heartbeat from core client for 30 sec - exiting 02:27:18 (6676): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5756, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:20:18 (5952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:20:25 (5952): No heartbeat from core client for 30 sec - exiting 04:03:24 (9996): No heartbeat from core client for 30 sec - exiting 04:03:31 (9996): No heartbeat from core client for 30 sec - exiting 04:03:32 (9996): No heartbeat from core client for 30 sec - exiting 04:03:33 (9996): No heartbeat from core client for 30 sec - exiting 04:03:34 (9996): No heartbeat from core client for 30 sec - exiting 04:03:35 (9996): No heartbeat from core client for 30 sec - exiting 04:03:36 (9996): No heartbeat from core client for 30 sec - exiting 04:03:37 (9996): No heartbeat from core client for 30 sec - exiting 04:03:38 (9996): No heartbeat from core client for 30 sec - exiting 04:03:39 (9996): No heartbeat from core client for 30 sec - exiting 04:03:40 (9996): No heartbeat from core client for 30 sec - exiting 04:03:41 (9996): No heartbeat from core client for 30 sec - exiting 04:03:42 (9996): No heartbeat from core client for 30 sec - exiting 04:03:43 (9996): No heartbeat from core client for 30 sec - exiting 04:03:44 (9996): No heartbeat from core client for 30 sec - exiting 04:03:45 (9996): No heartbeat from core client for 30 sec - exiting 04:03:46 (9996): No heartbeat from core client for 30 sec - exiting 04:03:47 (9996): No heartbeat from core client for 30 sec - exiting 04:03:48 (9996): No heartbeat from core client for 30 sec - exiting 04:03:49 (9996): No heartbeat from core client for 30 sec - exiting 04:03:50 (9996): No heartbeat from core client for 30 sec - exiting 04:03:51 (9996): No heartbeat from core client for 30 sec - exiting 04:03:52 (9996): No heartbeat from core client for 30 sec - exiting 04:03:53 (9996): No heartbeat from core client for 30 sec - exiting 04:03:54 (9996): No heartbeat from core client for 30 sec - exiting 04:04:01 (9996): No heartbeat from core client for 30 sec - exiting Signal 11 received, exiting... Called boinc_finish Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x771672D9 read attempt to address 0x40D771BF Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Nov 2013 20:28:48	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	518,400	1,644,261	3.1718
19 Nov 2013 07:58:38	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	492,480	1,556,044	3.1596
17 Nov 2013 05:32:51	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	466,560	1,468,902	3.1484
14 Nov 2013 20:48:11	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	440,640	1,380,330	3.1326
11 Nov 2013 08:06:19	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	414,720	1,297,502	3.1286
08 Nov 2013 05:36:28	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	388,800	1,216,758	3.1295
06 Nov 2013 08:32:29	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	362,880	1,132,826	3.1218
02 Nov 2013 12:19:29	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	336,960	1,050,097	3.1164
31 Oct 2013 06:18:37	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	311,040	965,776	3.1050
27 Oct 2013 13:10:26	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	285,120	882,166	3.0940
25 Oct 2013 05:36:42	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	259,200	794,788	3.0663
24 Oct 2013 21:52:46	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	233,280	720,132	3.0870
21 Oct 2013 23:08:44	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	207,360	638,286	3.0782
20 Oct 2013 11:20:34	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	181,440	552,053	3.0426
16 Oct 2013 23:03:20	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	155,520	469,996	3.0221
13 Oct 2013 23:02:07	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	129,600	386,173	2.9797
10 Oct 2013 09:21:29	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	103,680	312,294	3.0121
09 Oct 2013 05:56:16	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	77,760	238,700	3.0697
08 Oct 2013 02:45:29	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	51,840	165,882	3.1999
06 Oct 2013 15:46:22	895306	16039672	hadcm3n_oaph_1900_40_008469000_0	25,920	80,821	3.1181