Task 15925966

Name	hadcm3n_3agg_1980_40_008317853_3
Workunit	8468988
Created	19 Aug 2013, 7:59:14 UTC
Sent	19 Aug 2013, 7:59:27 UTC
Report deadline	18 Nov 2013, 15:26:38 UTC
Received	29 Oct 2013, 8:38:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1245167
Run time	5 days 6 hours 25 min 36 sec
CPU time	5 days 2 hours 48 min 5 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.67 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 08:34:35 (4672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:39 (4672): No heartbeat from core client for 30 sec - exiting 08:34:40 (4672): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 08:23:50 (1152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4000, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5020, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5020, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4292, iMonCtr=1 Model crash detected, will try to restart... 08:42:55 (1272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 14:48:43 (3472): No heartbeat from core client for 30 sec - exiting 14:48:44 (3472): No heartbeat from core client for 30 sec - exiting 14:48:45 (3472): No heartbeat from core client for 30 sec - exiting 14:48:46 (3472): No heartbeat from core client for 30 sec - exiting 14:48:47 (3472): No heartbeat from core client for 30 sec - exiting 14:48:48 (3472): No heartbeat from core client for 30 sec - exiting 14:48:49 (3472): No heartbeat from core client for 30 sec - exiting 14:48:51 (3472): No heartbeat from core client for 30 sec - exiting 14:48:52 (3472): No heartbeat from core client for 30 sec - exiting 14:48:53 (3472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 08:28:02 (5116): No heartbeat from core client for 30 sec - exiting 08:28:03 (5116): No heartbeat from core client for 30 sec - exiting 08:28:04 (5116): No heartbeat from core client for 30 sec - exiting 08:28:05 (5116): No heartbeat from core client for 30 sec - exiting 08:28:06 (5116): No heartbeat from core client for 30 sec - exiting 08:28:07 (5116): No heartbeat from core client for 30 sec - exiting 08:28:08 (5116): No heartbeat from core client for 30 sec - exiting 08:28:09 (5116): No heartbeat from core client for 30 sec - exiting 08:28:10 (5116): No heartbeat from core client for 30 sec - exiting 08:28:12 (5116): No heartbeat from core client for 30 sec - exiting 08:28:13 (5116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 15:19:07 (4780): No heartbeat from core client for 30 sec - exiting 15:19:08 (4780): No heartbeat from core client for 30 sec - exiting 15:19:09 (4780): No heartbeat from core client for 30 sec - exiting 15:19:10 (4780): No heartbeat from core client for 30 sec - exiting 15:19:11 (4780): No heartbeat from core client for 30 sec - exiting 15:19:12 (4780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:39:23 (4748): No heartbeat from core client for 30 sec - exiting 08:39:24 (4748): No heartbeat from core client for 30 sec - exiting 08:39:25 (4748): No heartbeat from core client for 30 sec - exiting 08:39:26 (4748): No heartbeat from core client for 30 sec - exiting 08:39:27 (4748): No heartbeat from core client for 30 sec - exiting 08:39:28 (4748): No heartbeat from core client for 30 sec - exiting 08:39:29 (4748): No heartbeat from core client for 30 sec - exiting 08:39:31 (4748): No heartbeat from core client for 30 sec - exiting 08:39:32 (4748): No heartbeat from core client for 30 sec - exiting 08:39:33 (4748): No heartbeat from core client for 30 sec - exiting 08:39:34 (4748): No heartbeat from core client for 30 sec - exiting 08:39:35 (4748): No heartbeat from core client for 30 sec - exiting 08:39:36 (4748): No heartbeat from core client for 30 sec - exiting 08:39:37 (4748): No heartbeat from core client for 30 sec - exiting 08:39:38 (4748): No heartbeat from core client for 30 sec - exiting 08:39:39 (4748): No heartbeat from core client for 30 sec - exiting 08:39:40 (4748): No heartbeat from core client for 30 sec - exiting 08:39:41 (4748): No heartbeat from core client for 30 sec - exiting 08:39:43 (4748): No heartbeat from core client for 30 sec - exiting 08:39:44 (4748): No heartbeat from core client for 30 sec - exiting 08:39:45 (4748): No heartbeat from core client for 30 sec - exiting 08:39:46 (4748): No heartbeat from core client for 30 sec - exiting 08:39:47 (4748): No heartbeat from core client for 30 sec - exiting 08:39:48 (4748): No heartbeat from core client for 30 sec - exiting 08:39:49 (4748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 11:06:39 (5960): No heartbeat from core client for 30 sec - exiting 11:06:40 (5960): No heartbeat from core client for 30 sec - exiting 11:06:41 (5960): No heartbeat from core client for 30 sec - exiting 11:06:42 (5960): No heartbeat from core client for 30 sec - exiting 11:06:43 (5960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 08:36:08 (3556): No heartbeat from core client for 30 sec - exiting 08:36:09 (3556): No heartbeat from core client for 30 sec - exiting 08:36:10 (3556): No heartbeat from core client for 30 sec - exiting 08:36:11 (3556): No heartbeat from core client for 30 sec - exiting 08:36:13 (3556): No heartbeat from core client for 30 sec - exiting 08:36:14 (3556): No heartbeat from core client for 30 sec - exiting 08:36:15 (3556): No heartbeat from core client for 30 sec - exiting 08:36:16 (3556): No heartbeat from core client for 30 sec - exiting 08:36:17 (3556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 09:05:13 (1476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 08:32:14 (4924): No heartbeat from core client for 30 sec - exiting 08:32:15 (4924): No heartbeat from core client for 30 sec - exiting 08:32:16 (4924): No heartbeat from core client for 30 sec - exiting 08:32:17 (4924): No heartbeat from core client for 30 sec - exiting 08:32:18 (4924): No heartbeat from core client for 30 sec - exiting 08:32:19 (4924): No heartbeat from core client for 30 sec - exiting 08:32:20 (4924): No heartbeat from core client for 30 sec - exiting 08:32:21 (4924): No heartbeat from core client for 30 sec - exiting 08:32:22 (4924): No heartbeat from core client for 30 sec - exiting 08:32:23 (4924): No heartbeat from core client for 30 sec - exiting 08:32:25 (4924): No heartbeat from core client for 30 sec - exiting 08:32:26 (4924): No heartbeat from core client for 30 sec - exiting 08:32:27 (4924): No heartbeat from core client for 30 sec - exiting 08:32:28 (4924): No heartbeat from core client for 30 sec - exiting 08:32:29 (4924): No heartbeat from core client for 30 sec - exiting 08:32:30 (4924): No heartbeat from core client for 30 sec - exiting 08:32:31 (4924): No heartbeat from core client for 30 sec - exiting 08:32:32 (4924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:54:20 (1340): No heartbeat from core client for 30 sec - exiting 08:54:21 (1340): No heartbeat from core client for 30 sec - exiting 08:54:22 (1340): No heartbeat from core client for 30 sec - exiting 08:54:23 (1340): No heartbeat from core client for 30 sec - exiting 08:54:24 (1340): No heartbeat from core client for 30 sec - exiting 08:54:26 (1340): No heartbeat from core client for 30 sec - exiting 08:54:27 (1340): No heartbeat from core client for 30 sec - exiting 08:54:28 (1340): No heartbeat from core client for 30 sec - exiting 08:54:29 (1340): No heartbeat from core client for 30 sec - exiting 08:54:30 (1340): No heartbeat from core client for 30 sec - exiting 08:54:31 (1340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7745FF6B write attempt to address 0x434DAD63 Engaging BOINC Windows Runtime Debugger... 08:40:09 (5672): No heartbeat from core client for 30 sec - exiting 08:40:10 (5672): No heartbeat from core client for 30 sec - exiting 08:40:11 (5672): No heartbeat from core client for 30 sec - exiting 08:40:12 (5672): No heartbeat from core client for 30 sec - exiting 08:40:13 (5672): No heartbeat from core client for 30 sec - exiting 08:40:14 (5672): No heartbeat from core client for 30 sec - exiting 08:40:16 (5672): No heartbeat from core client for 30 sec - exiting 08:40:17 (5672): No heartbeat from core client for 30 sec - exiting 08:40:18 (5672): No heartbeat from core client for 30 sec - exiting 08:40:19 (5672): No heartbeat from core client for 30 sec - exiting 08:40:20 (5672): No heartbeat from core client for 30 sec - exiting 08:40:21 (5672): No heartbeat from core client for 30 sec - exiting 08:40:22 (5672): No heartbeat from core client for 30 sec - exiting 08:40:23 (5672): No heartbeat from core client for 30 sec - exiting 08:40:24 (5672): No heartbeat from core client for 30 sec - exiting 08:40:25 (5672): No heartbeat from core client for 30 sec - exiting 08:40:26 (5672): No heartbeat from core client for 30 sec - exiting 08:40:27 (5672): No heartbeat from core client for 30 sec - exiting 08:40:28 (5672): No heartbeat from core client for 30 sec - exiting 08:40:30 (5672): No heartbeat from core client for 30 sec - exiting 08:40:31 (5672): No heartbeat from core client for 30 sec - exiting 08:40:32 (5672): No heartbeat from core client for 30 sec - exiting 08:40:33 (5672): No heartbeat from core client for 30 sec - exiting 08:40:34 (5672): No heartbeat from core client for 30 sec - exiting 08:40:35 (5672): No heartbeat from core client for 30 sec - exiting 08:40:36 (5672): No heartbeat from core client for 30 sec - exiting 08:40:37 (5672): No heartbeat from core client for 30 sec - exiting 08:40:38 (5672): No heartbeat from core client for 30 sec - exiting 08:40:39 (5672): No heartbeat from core client for 30 sec - exiting 08:40:40 (5672): No heartbeat from core client for 30 sec - exiting 08:40:42 (5672): No heartbeat from core client for 30 sec - exiting 08:40:43 (5672): No heartbeat from core client for 30 sec - exiting 08:40:44 (5672): No heartbeat from core client for 30 sec - exiting 08:40:45 (5672): No heartbeat from core client for 30 sec - exiting 08:40:46 (5672): No heartbeat from core client for 30 sec - exiting 08:40:47 (5672): No heartbeat from core client for 30 sec - exiting 08:40:48 (5672): No heartbeat from core client for 30 sec - exiting 08:40:49 (5672): No heartbeat from core client for 30 sec - exiting 08:40:50 (5672): No heartbeat from core client for 30 sec - exiting 08:40:51 (5672): No heartbeat from core client for 30 sec - exiting 08:40:52 (5672): No heartbeat from core client for 30 sec - exiting 08:40:54 (5672): No heartbeat from core client for 30 sec - exiting 08:40:55 (5672): No heartbeat from core client for 30 sec - exiting 08:40:56 (5672): No heartbeat from core client for 30 sec - exiting 08:40:57 (5672): No heartbeat from core client for 30 sec - exiting 08:40:58 (5672): No heartbeat from core client for 30 sec - exiting 08:40:59 (5672): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76F77383 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3agg_1980_40_008317853/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish 08:41:00 (5672): No heartbeat from core client for 30 sec - exiting 08:41:01 (5672): No heartbeat from core client for 30 sec - exiting </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Oct 2013 14:41:15	1245167	15925966	hadcm3n_3agg_1980_40_008317853_3	259,200	439,356	1.6950
24 Oct 2013 07:38:42	1245167	15925966	hadcm3n_3agg_1980_40_008317853_3	233,280	393,832	1.6882
17 Oct 2013 09:08:35	1245167	15925966	hadcm3n_3agg_1980_40_008317853_3	207,360	348,623	1.6812
08 Oct 2013 12:18:52	1245167	15925966	hadcm3n_3agg_1980_40_008317853_3	181,440	302,957	1.6697
04 Oct 2013 10:32:33	1245167	15925966	hadcm3n_3agg_1980_40_008317853_3	155,520	257,880	1.6582
02 Oct 2013 07:28:44	1245167	15925966	hadcm3n_3agg_1980_40_008317853_3	129,600	214,456	1.6548
25 Sep 2013 09:20:51	1245167	15925966	hadcm3n_3agg_1980_40_008317853_3	103,680	171,852	1.6575
10 Sep 2013 08:34:20	1245167	15925966	hadcm3n_3agg_1980_40_008317853_3	77,760	129,286	1.6626
05 Sep 2013 08:17:58	1245167	15925966	hadcm3n_3agg_1980_40_008317853_3	51,840	86,030	1.6595
29 Aug 2013 12:01:23	1245167	15925966	hadcm3n_3agg_1980_40_008317853_3	25,920	42,606	1.6438