Task 15555721

Name	hadcm3n_zkkd_1880_40_008248514_4
Workunit	8403638
Created	23 Jan 2013, 16:19:53 UTC
Sent	23 Jan 2013, 16:19:54 UTC
Report deadline	24 Apr 2013, 23:47:05 UTC
Received	16 May 2013, 20:29:20 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1061339
Run time	26 days 13 hours 1 min 55 sec
CPU time	25 days 14 hours 58 min 29 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	1.88 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1256, iMonCtr=1 Model crash detected, will try to restart... 12:59:41 (5352): No heartbeat from core client for 30 sec - exiting 12:59:43 (5352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:07:13 (4352): No heartbeat from core client for 30 sec - exiting 15:07:14 (4352): No heartbeat from core client for 30 sec - exiting 15:07:15 (4352): No heartbeat from core client for 30 sec - exiting 15:07:16 (4352): No heartbeat from core client for 30 sec - exiting 15:07:17 (4352): No heartbeat from core client for 30 sec - exiting 15:07:19 (4352): No heartbeat from core client for 30 sec - exiting 15:07:20 (4352): No heartbeat from core client for 30 sec - exiting 15:07:21 (4352): No heartbeat from core client for 30 sec - exiting 15:07:22 (4352): No heartbeat from core client for 30 sec - exiting 15:07:23 (4352): No heartbeat from core client for 30 sec - exiting 15:07:24 (4352): No heartbeat from core client for 30 sec - exiting 15:07:25 (4352): No heartbeat from core client for 30 sec - exiting 15:07:26 (4352): No heartbeat from core client for 30 sec - exiting 15:07:27 (4352): No heartbeat from core client for 30 sec - exiting 15:07:28 (4352): No heartbeat from core client for 30 sec - exiting 15:07:29 (4352): No heartbeat from core client for 30 sec - exiting 15:07:31 (4352): No heartbeat from core client for 30 sec - exiting 15:07:32 (4352): No heartbeat from core client for 30 sec - exiting 15:07:33 (4352): No heartbeat from core client for 30 sec - exiting 15:07:34 (4352): No heartbeat from core client for 30 sec - exiting 15:07:35 (4352): No heartbeat from core client for 30 sec - exiting 15:07:36 (4352): No heartbeat from core client for 30 sec - exiting 15:07:37 (4352): No heartbeat from core client for 30 sec - exiting 15:07:38 (4352): No heartbeat from core client for 30 sec - exiting 15:07:39 (4352): No heartbeat from core client for 30 sec - exiting 15:07:40 (4352): No heartbeat from core client for 30 sec - exiting 15:07:42 (4352): No heartbeat from core client for 30 sec - exiting 15:07:43 (4352): No heartbeat from core client for 30 sec - exiting 15:07:44 (4352): No heartbeat from core client for 30 sec - exiting 15:07:45 (4352): No heartbeat from core client for 30 sec - exiting 15:07:46 (4352): No heartbeat from core client for 30 sec - exiting 15:07:47 (4352): No heartbeat from core client for 30 sec - exiting 15:07:48 (4352): No heartbeat from core client for 30 sec - exiting 15:07:49 (4352): No heartbeat from core client for 30 sec - exiting 15:07:50 (4352): No heartbeat from core client for 30 sec - exiting 15:07:51 (4352): No heartbeat from core client for 30 sec - exiting 15:07:52 (4352): No heartbeat from core client for 30 sec - exiting 15:07:54 (4352): No heartbeat from core client for 30 sec - exiting 15:07:55 (4352): No heartbeat from core client for 30 sec - exiting 15:07:56 (4352): No heartbeat from core client for 30 sec - exiting 15:07:57 (4352): No heartbeat from core client for 30 sec - exiting 15:07:58 (4352): No heartbeat from core client for 30 sec - exiting 15:07:59 (4352): No heartbeat from core client for 30 sec - exiting 15:08:00 (4352): No heartbeat from core client for 30 sec - exiting 15:08:01 (4352): No heartbeat from core client for 30 sec - exiting 15:08:02 (4352): No heartbeat from core client for 30 sec - exiting 15:08:03 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:08:04 (4352): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:42:49 (5404): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:25:42 (4720): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 07:25:43 (4720): No heartbeat from core client for 30 sec - exiting 07:25:44 (4720): No heartbeat from core client for 30 sec - exiting 07:25:46 (4720): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4568, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4300, iMonCtr=1 Model crash detected, will try to restart... 06:50:51 (5288): No heartbeat from core client for 30 sec - exiting 06:50:52 (5288): No heartbeat from core client for 30 sec - exiting 06:50:53 (5288): No heartbeat from core client for 30 sec - exiting 06:50:54 (5288): No heartbeat from core client for 30 sec - exiting 06:50:55 (5288): No heartbeat from core client for 30 sec - exiting 06:50:56 (5288): No heartbeat from core client for 30 sec - exiting 06:50:57 (5288): No heartbeat from core client for 30 sec - exiting 06:50:59 (5288): No heartbeat from core client for 30 sec - exiting 06:51:00 (5288): No heartbeat from core client for 30 sec - exiting 06:51:01 (5288): No heartbeat from core client for 30 sec - exiting 06:51:02 (5288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:36:53 (4328): No heartbeat from core client for 30 sec - exiting 10:36:54 (4328): No heartbeat from core client for 30 sec - exiting 10:36:55 (4328): No heartbeat from core client for 30 sec - exiting 10:36:56 (4328): No heartbeat from core client for 30 sec - exiting 10:36:57 (4328): No heartbeat from core client for 30 sec - exiting 10:36:58 (4328): No heartbeat from core client for 30 sec - exiting 10:36:59 (4328): No heartbeat from core client for 30 sec - exiting 10:37:01 (4328): No heartbeat from core client for 30 sec - exiting 10:37:02 (4328): No heartbeat from core client for 30 sec - exiting 10:37:03 (4328): No heartbeat from core client for 30 sec - exiting 10:37:04 (4328): No heartbeat from core client for 30 sec - exiting 10:37:05 (4328): No heartbeat from core client for 30 sec - exiting 10:37:06 (4328): No heartbeat from core client for 30 sec - exiting 10:37:07 (4328): No heartbeat from core client for 30 sec - exiting 10:37:08 (4328): No heartbeat from core client for 30 sec - exiting 10:37:09 (4328): No heartbeat from core client for 30 sec - exiting 10:37:10 (4328): No heartbeat from core client for 30 sec - exiting 10:37:11 (4328): No heartbeat from core client for 30 sec - exiting 10:37:13 (4328): No heartbeat from core client for 30 sec - exiting 10:37:14 (4328): No heartbeat from core client for 30 sec - exiting 10:37:15 (4328): No heartbeat from core client for 30 sec - exiting 10:37:16 (4328): No heartbeat from core client for 30 sec - exiting 10:37:17 (4328): No heartbeat from core client for 30 sec - exiting 10:37:18 (4328): No heartbeat from core client for 30 sec - exiting 10:37:19 (4328): No heartbeat from core client for 30 sec - exiting 10:37:20 (4328): No heartbeat from core client for 30 sec - exiting 10:37:21 (4328): No heartbeat from core client for 30 sec - exiting 10:37:22 (4328): No heartbeat from core client for 30 sec - exiting 10:37:24 (4328): No heartbeat from core client for 30 sec - exiting 10:37:25 (4328): No heartbeat from core client for 30 sec - exiting 10:37:26 (4328): No heartbeat from core client for 30 sec - exiting 10:37:27 (4328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:17:30 (4564): No heartbeat from core client for 30 sec - exiting 17:17:31 (4564): No heartbeat from core client for 30 sec - exiting 17:17:32 (4564): No heartbeat from core client for 30 sec - exiting 17:17:33 (4564): No heartbeat from core client for 30 sec - exiting 17:17:34 (4564): No heartbeat from core client for 30 sec - exiting 17:17:35 (4564): No heartbeat from core client for 30 sec - exiting 17:17:36 (4564): No heartbeat from core client for 30 sec - exiting 17:17:38 (4564): No heartbeat from core client for 30 sec - exiting 17:17:39 (4564): No heartbeat from core client for 30 sec - exiting 17:17:40 (4564): No heartbeat from core client for 30 sec - exiting 17:17:41 (4564): No heartbeat from core client for 30 sec - exiting 17:17:42 (4564): No heartbeat from core client for 30 sec - exiting 17:17:43 (4564): No heartbeat from core client for 30 sec - exiting 17:17:44 (4564): No heartbeat from core client for 30 sec - exiting 17:17:45 (4564): No heartbeat from core client for 30 sec - exiting 17:17:46 (4564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:48:47 (5328): No heartbeat from core client for 30 sec - exiting 08:48:49 (5328): No heartbeat from core client for 30 sec - exiting 08:48:50 (5328): No heartbeat from core client for 30 sec - exiting 08:48:51 (5328): No heartbeat from core client for 30 sec - exiting 08:48:52 (5328): No heartbeat from core client for 30 sec - exiting 08:48:53 (5328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:48:54 (5328): No heartbeat from core client for 30 sec - exiting 07:26:59 (5076): No heartbeat from core client for 30 sec - exiting 07:27:00 (5076): No heartbeat from core client for 30 sec - exiting 07:27:01 (5076): No heartbeat from core client for 30 sec - exiting 07:27:02 (5076): No heartbeat from core client for 30 sec - exiting 07:27:03 (5076): No heartbeat from core client for 30 sec - exiting 07:27:04 (5076): No heartbeat from core client for 30 sec - exiting 07:27:06 (5076): No heartbeat from core client for 30 sec - exiting 07:27:07 (5076): No heartbeat from core client for 30 sec - exiting 07:27:08 (5076): No heartbeat from core client for 30 sec - exiting 07:27:09 (5076): No heartbeat from core client for 30 sec - exiting 07:27:10 (5076): No heartbeat from core client for 30 sec - exiting 07:27:11 (5076): No heartbeat from core client for 30 sec - exiting 07:27:12 (5076): No heartbeat from core client for 30 sec - exiting 07:27:13 (5076): No heartbeat from core client for 30 sec - exiting 07:27:14 (5076): No heartbeat from core client for 30 sec - exiting 07:27:15 (5076): No heartbeat from core client for 30 sec - exiting 07:27:17 (5076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:41:25 (4400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:41:27 (4400): No heartbeat from core client for 30 sec - exiting 00:17:00 (5056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:17:02 (5056): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 06:51:19 (5156): No heartbeat from core client for 30 sec - exiting 06:51:20 (5156): No heartbeat from core client for 30 sec - exiting 06:51:22 (5156): No heartbeat from core client for 30 sec - exiting 06:51:23 (5156): No heartbeat from core client for 30 sec - exiting 06:51:24 (5156): No heartbeat from core client for 30 sec - exiting 06:51:25 (5156): No heartbeat from core client for 30 sec - exiting 06:51:26 (5156): No heartbeat from core client for 30 sec - exiting 06:51:27 (5156): No heartbeat from core client for 30 sec - exiting 06:51:28 (5156): No heartbeat from core client for 30 sec - exiting 06:51:29 (5156): No heartbeat from core client for 30 sec - exiting 06:51:30 (5156): No heartbeat from core client for 30 sec - exiting 06:51:31 (5156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77335EAB read attempt to address 0x404885F8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7777A5D5 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zkkd_1880_40_008248514/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 May 2013 06:50:40	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	1,036,800	2,213,167	2.1346
04 May 2013 06:50:40	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	1,010,880	2,163,393	2.1401
04 May 2013 06:50:40	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	984,960	2,112,482	2.1447
30 Apr 2013 02:53:36	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	959,040	2,052,979	2.1407
29 Apr 2013 08:57:47	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	933,120	1,994,936	2.1379
28 Apr 2013 14:20:29	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	907,200	1,936,666	2.1348
27 Apr 2013 20:36:38	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	881,280	1,878,616	2.1317
27 Apr 2013 04:31:42	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	855,360	1,822,436	2.1306
27 Apr 2013 03:31:25	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	829,440	1,770,506	2.1346
27 Apr 2013 03:31:25	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	803,520	1,717,820	2.1379
25 Apr 2013 03:05:06	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	777,600	1,663,922	2.1398
24 Apr 2013 23:03:32	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	751,680	1,611,874	2.1444
24 Apr 2013 23:03:32	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	725,760	1,561,059	2.1509
24 Apr 2013 23:03:32	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	699,840	1,510,161	2.1579
28 Mar 2013 21:08:20	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	673,920	1,455,523	2.1598
27 Mar 2013 16:15:45	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	648,000	1,400,892	2.1619
26 Mar 2013 05:19:46	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	622,080	1,346,469	2.1645
24 Mar 2013 18:47:38	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	596,160	1,292,599	2.1682
23 Mar 2013 17:47:16	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	570,240	1,238,807	2.1724
21 Mar 2013 20:54:04	1061339	15555721	hadcm3n_zkkd_1880_40_008248514_4	544,320	1,184,459	2.1760