Task 13606566

Name	hadcm3n_ybf8_1940_40_007538488_1
Workunit	7735720
Created	5 Nov 2011, 21:15:13 UTC
Sent	9 Nov 2011, 20:54:49 UTC
Report deadline	9 Feb 2012, 4:22:00 UTC
Received	1 Jan 2012, 18:16:26 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1021113
Run time	18 days 22 hours 15 min 1 sec
CPU time	18 days 0 hours 30 min 34 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7476, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5904, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4500, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3488, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1184, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4928, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5380, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5132, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4612, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5604, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:09:43 (4368): No heartbeat from core client for 30 sec - exiting 09:09:44 (4368): No heartbeat from core client for 30 sec - exiting 09:09:45 (4368): No heartbeat from core client for 30 sec - exiting 09:09:46 (4368): No heartbeat from core client for 30 sec - exiting 09:09:47 (4368): No heartbeat from core client for 30 sec - exiting 09:09:48 (4368): No heartbeat from core client for 30 sec - exiting 09:09:49 (4368): No heartbeat from core client for 30 sec - exiting 09:09:50 (4368): No heartbeat from core client for 30 sec - exiting 09:09:51 (4368): No heartbeat from core client for 30 sec - exiting 09:09:52 (4368): No heartbeat from core client for 30 sec - exiting 09:09:53 (4368): No heartbeat from core client for 30 sec - exiting 09:09:54 (4368): No heartbeat from core client for 30 sec - exiting 09:09:55 (4368): No heartbeat from core client for 30 sec - exiting 09:09:56 (4368): No heartbeat from core client for 30 sec - exiting 09:09:57 (4368): No heartbeat from core client for 30 sec - exiting 09:09:58 (4368): No heartbeat from core client for 30 sec - exiting 09:09:59 (4368): No heartbeat from core client for 30 sec - exiting 09:10:00 (4368): No heartbeat from core client for 30 sec - exiting 09:10:01 (4368): No heartbeat from core client for 30 sec - exiting 09:10:02 (4368): No heartbeat from core client for 30 sec - exiting 09:10:03 (4368): No heartbeat from core client for 30 sec - exiting 09:10:04 (4368): No heartbeat from core client for 30 sec - exiting 09:10:05 (4368): No heartbeat from core client for 30 sec - exiting 09:10:06 (4368): No heartbeat from core client for 30 sec - exiting 09:10:07 (4368): No heartbeat from core client for 30 sec - exiting 09:10:08 (4368): No heartbeat from core client for 30 sec - exiting 09:10:09 (4368): No heartbeat from core client for 30 sec - exiting 09:10:10 (4368): No heartbeat from core client for 30 sec - exiting 09:10:11 (4368): No heartbeat from core client for 30 sec - exiting 09:10:12 (4368): No heartbeat from core client for 30 sec - exiting 09:10:13 (4368): No heartbeat from core client for 30 sec - exiting 09:10:14 (4368): No heartbeat from core client for 30 sec - exiting 09:10:15 (4368): No heartbeat from core client for 30 sec - exiting 09:10:16 (4368): No heartbeat from core client for 30 sec - exiting 09:10:17 (4368): No heartbeat from core client for 30 sec - exiting 09:10:18 (4368): No heartbeat from core client for 30 sec - exiting 09:10:19 (4368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:10:20 (4368): No heartbeat from core client for 30 sec - exiting 09:10:21 (4368): No heartbeat from core client for 30 sec - exiting 09:10:22 (4368): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5392, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:13:05 (5772): No heartbeat from core client for 30 sec - exiting 10:13:06 (5772): No heartbeat from core client for 30 sec - exiting 10:13:07 (5772): No heartbeat from core client for 30 sec - exiting 10:13:08 (5772): No heartbeat from core client for 30 sec - exiting 10:13:09 (5772): No heartbeat from core client for 30 sec - exiting 10:13:10 (5772): No heartbeat from core client for 30 sec - exiting 10:13:11 (5772): No heartbeat from core client for 30 sec - exiting 10:13:12 (5772): No heartbeat from core client for 30 sec - exiting 10:13:13 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6444, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4168, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5700, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1608, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6708, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5300, iMonCtr=1 Model crash detected, will try to restart... 16:16:56 (5788): No heartbeat from core client for 30 sec - exiting 16:16:57 (5788): No heartbeat from core client for 30 sec - exiting 16:16:58 (5788): No heartbeat from core client for 30 sec - exiting 16:16:59 (5788): No heartbeat from core client for 30 sec - exiting 16:17:00 (5788): No heartbeat from core client for 30 sec - exiting 16:17:01 (5788): No heartbeat from core client for 30 sec - exiting 16:17:02 (5788): No heartbeat from core client for 30 sec - exiting 16:17:03 (5788): No heartbeat from core client for 30 sec - exiting 16:17:04 (5788): No heartbeat from core client for 30 sec - exiting 16:17:05 (5788): No heartbeat from core client for 30 sec - exiting 16:17:06 (5788): No heartbeat from core client for 30 sec - exiting 16:17:07 (5788): No heartbeat from core client for 30 sec - exiting 16:17:08 (5788): No heartbeat from core client for 30 sec - exiting 16:17:09 (5788): No heartbeat from core client for 30 sec - exiting 16:17:10 (5788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6644, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5856, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4168, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4168, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4168, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5928, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5268, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4876, iMonCtr=1 Model crash detected, will try to restart... 12:33:44 (5296): No heartbeat from core client for 30 sec - exiting 12:33:45 (5296): No heartbeat from core client for 30 sec - exiting 12:33:46 (5296): No heartbeat from core client for 30 sec - exiting 12:33:47 (5296): No heartbeat from core client for 30 sec - exiting 12:33:48 (5296): No heartbeat from core client for 30 sec - exiting 12:33:49 (5296): No heartbeat from core client for 30 sec - exiting 12:33:50 (5296): No heartbeat from core client for 30 sec - exiting 12:33:51 (5296): No heartbeat from core client for 30 sec - exiting 12:33:52 (5296): No heartbeat from core client for 30 sec - exiting 12:33:53 (5296): No heartbeat from core client for 30 sec - exiting 12:33:54 (5296): No heartbeat from core client for 30 sec - exiting 12:33:55 (5296): No heartbeat from core client for 30 sec - exiting 12:33:56 (5296): No heartbeat from core client for 30 sec - exiting 12:33:57 (5296): No heartbeat from core client for 30 sec - exiting 12:33:58 (5296): No heartbeat from core client for 30 sec - exiting 12:33:59 (5296): No heartbeat from core client for 30 sec - exiting 12:34:00 (5296): No heartbeat from core client for 30 sec - exiting 12:34:01 (5296): No heartbeat from core client for 30 sec - exiting 12:34:02 (5296): No heartbeat from core client for 30 sec - exiting 12:34:03 (5296): No heartbeat from core client for 30 sec - exiting 12:34:04 (5296): No heartbeat from core client for 30 sec - exiting 12:34:05 (5296): No heartbeat from core client for 30 sec - exiting 12:34:06 (5296): No heartbeat from core client for 30 sec - exiting 12:34:07 (5296): No heartbeat from core client for 30 sec - exiting 12:34:08 (5296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5612, iMonCtr=1 Model crash detected, will try to restart... 10:40:33 (5660): No heartbeat from core client for 30 sec - exiting 10:40:35 (5660): No heartbeat from core client for 30 sec - exiting 10:40:36 (5660): No heartbeat from core client for 30 sec - exiting 10:40:37 (5660): No heartbeat from core client for 30 sec - exiting 10:40:38 (5660): No heartbeat from core client for 30 sec - exiting 10:40:39 (5660): No heartbeat from core client for 30 sec - exiting 10:40:40 (5660): No heartbeat from core client for 30 sec - exiting 10:40:41 (5660): No heartbeat from core client for 30 sec - exiting 10:40:42 (5660): No heartbeat from core client for 30 sec - exiting 10:40:43 (5660): No heartbeat from core client for 30 sec - exiting 10:40:44 (5660): No heartbeat from core client for 30 sec - exiting 10:40:45 (5660): No heartbeat from core client for 30 sec - exiting 10:40:46 (5660): No heartbeat from core client for 30 sec - exiting 10:40:47 (5660): No heartbeat from core client for 30 sec - exiting 10:40:48 (5660): No heartbeat from core client for 30 sec - exiting 10:40:49 (5660): No heartbeat from core client for 30 sec - exiting 10:40:50 (5660): No heartbeat from core client for 30 sec - exiting 10:40:51 (5660): No heartbeat from core client for 30 sec - exiting 10:40:52 (5660): No heartbeat from core client for 30 sec - exiting 10:40:53 (5660): No heartbeat from core client for 30 sec - exiting 10:40:54 (5660): No heartbeat from core client for 30 sec - exiting 10:40:55 (5660): No heartbeat from core client for 30 sec - exiting 10:40:56 (5660): No heartbeat from core client for 30 sec - exiting 10:40:57 (5660): No heartbeat from core client for 30 sec - exiting 10:40:58 (5660): No heartbeat from core client for 30 sec - exiting 10:40:59 (5660): No heartbeat from core client for 30 sec - exiting 10:41:00 (5660): No heartbeat from core client for 30 sec - exiting 10:41:01 (5660): No heartbeat from core client for 30 sec - exiting 10:41:02 (5660): No heartbeat from core client for 30 sec - exiting 10:41:03 (5660): No heartbeat from core client for 30 sec - exiting 10:41:04 (5660): No heartbeat from core client for 30 sec - exiting 10:41:05 (5660): No heartbeat from core client for 30 sec - exiting 10:41:06 (5660): No heartbeat from core client for 30 sec - exiting 10:41:07 (5660): No heartbeat from core client for 30 sec - exiting 10:41:08 (5660): No heartbeat from core client for 30 sec - exiting 10:41:09 (5660): No heartbeat from core client for 30 sec - exiting 10:41:10 (5660): No heartbeat from core client for 30 sec - exiting 10:41:11 (5660): No heartbeat from core client for 30 sec - exiting 10:41:12 (5660): No heartbeat from core client for 30 sec - exiting 10:41:13 (5660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6904, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7076, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3988, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1112, iMonCtr=1 Model crash detected, will try to restart... 11:33:05 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77CB3A93 read attempt to address 0x40CB1C2B Engaging BOINC Windows Runtime Debugger... Cannot serialize file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ybf8_1940_40_007538488/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Jan 2012 18:43:27	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	777,600	1,557,029	2.0024
31 Dec 2011 16:28:09	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	751,680	1,505,575	2.0029
29 Dec 2011 19:29:09	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	725,760	1,453,166	2.0023
27 Dec 2011 19:04:46	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	699,840	1,399,757	2.0001
26 Dec 2011 20:13:51	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	673,920	1,346,904	1.9986
25 Dec 2011 18:04:21	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	648,000	1,294,478	1.9977
24 Dec 2011 21:55:59	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	622,080	1,244,057	1.9998
23 Dec 2011 18:33:13	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	596,160	1,192,210	1.9998
22 Dec 2011 19:02:58	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	570,240	1,138,901	1.9972
21 Dec 2011 12:23:48	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	544,320	1,087,053	1.9971
19 Dec 2011 15:19:15	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	518,400	1,034,666	1.9959
16 Dec 2011 16:53:54	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	492,480	982,524	1.9951
13 Dec 2011 15:33:15	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	466,560	929,418	1.9921
11 Dec 2011 10:57:18	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	440,640	876,673	1.9895
10 Dec 2011 09:26:13	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	414,720	824,999	1.9893
08 Dec 2011 15:04:44	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	388,800	773,393	1.9892
06 Dec 2011 13:51:26	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	362,880	721,766	1.9890
05 Dec 2011 14:37:45	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	336,960	669,036	1.9855
04 Dec 2011 11:15:32	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	311,040	615,368	1.9784
02 Dec 2011 20:32:14	1021113	13606566	hadcm3n_ybf8_1940_40_007538488_1	285,120	563,265	1.9755