Task 12953630

Name	hadcm3n_o0fo_1940_40_007264632_2
Workunit	7462872
Created	7 Jun 2011, 15:10:38 UTC
Sent	7 Jun 2011, 15:10:42 UTC
Report deadline	6 Sep 2011, 22:37:53 UTC
Received	20 Jul 2011, 12:29:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1036099
Run time	6 days 8 hours 54 min 21 sec
CPU time	5 days 20 hours 13 min 19 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 07:45:14 (7004): No heartbeat from core client for 30 sec - exiting 07:45:15 (7004): No heartbeat from core client for 30 sec - exiting 07:45:16 (7004): No heartbeat from core client for 30 sec - exiting 07:45:17 (7004): No heartbeat from core client for 30 sec - exiting 07:45:18 (7004): No heartbeat from core client for 30 sec - exiting 07:45:19 (7004): No heartbeat from core client for 30 sec - exiting 07:45:20 (7004): No heartbeat from core client for 30 sec - exiting 07:45:21 (7004): No heartbeat from core client for 30 sec - exiting 07:45:22 (7004): No heartbeat from core client for 30 sec - exiting 07:45:23 (7004): No heartbeat from core client for 30 sec - exiting 07:45:24 (7004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:17:19 (6784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:52:28 (5992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:07:21 (3984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:07:23 (3984): No heartbeat from core client for 30 sec - exiting 16:07:24 (3984): No heartbeat from core client for 30 sec - exiting 16:07:25 (3984): No heartbeat from core client for 30 sec - exiting 16:08:02 (7932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:28:50 (6176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:28:51 (6176): No heartbeat from core client for 30 sec - exiting 17:28:52 (6176): No heartbeat from core client for 30 sec - exiting 17:28:53 (6176): No heartbeat from core client for 30 sec - exiting 17:30:22 (2352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:44:09 (7600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:44:50 (7416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:53 (7148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:59:59 (8248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:00:00 (8248): No heartbeat from core client for 30 sec - exiting 18:37:09 (8936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:37:46 (6892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:37:47 (6892): No heartbeat from core client for 30 sec - exiting 18:37:48 (6892): No heartbeat from core client for 30 sec - exiting 11:14:53 (5480): No heartbeat from core client for 30 sec - exiting 11:14:54 (5480): No heartbeat from core client for 30 sec - exiting 11:14:55 (5480): No heartbeat from core client for 30 sec - exiting 11:14:56 (5480): No heartbeat from core client for 30 sec - exiting 11:14:57 (5480): No heartbeat from core client for 30 sec - exiting 11:14:58 (5480): No heartbeat from core client for 30 sec - exiting 11:14:59 (5480): No heartbeat from core client for 30 sec - exiting 11:15:01 (5480): No heartbeat from core client for 30 sec - exiting 11:15:02 (5480): No heartbeat from core client for 30 sec - exiting 11:15:03 (5480): No heartbeat from core client for 30 sec - exiting 11:15:04 (5480): No heartbeat from core client for 30 sec - exiting 11:15:05 (5480): No heartbeat from core client for 30 sec - exiting 11:15:06 (5480): No heartbeat from core client for 30 sec - exiting 11:15:07 (5480): No heartbeat from core client for 30 sec - exiting 11:15:08 (5480): No heartbeat from core client for 30 sec - exiting 11:15:09 (5480): No heartbeat from core client for 30 sec - exiting 11:15:10 (5480): No heartbeat from core client for 30 sec - exiting 11:15:11 (5480): No heartbeat from core client for 30 sec - exiting 11:15:13 (5480): No heartbeat from core client for 30 sec - exiting 11:15:14 (5480): No heartbeat from core client for 30 sec - exiting 11:15:15 (5480): No heartbeat from core client for 30 sec - exiting 11:15:16 (5480): No heartbeat from core client for 30 sec - exiting 11:15:17 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:15:18 (5480): No heartbeat from core client for 30 sec - exiting 11:15:19 (5480): No heartbeat from core client for 30 sec - exiting 11:15:20 (5480): No heartbeat from core client for 30 sec - exiting 11:15:21 (5480): No heartbeat from core client for 30 sec - exiting 11:15:22 (5480): No heartbeat from core client for 30 sec - exiting 11:15:23 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:06:49 (3272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:43:48 (2164): No heartbeat from core client for 30 sec - exiting 07:43:49 (2164): No heartbeat from core client for 30 sec - exiting 07:43:51 (2164): No heartbeat from core client for 30 sec - exiting 07:43:52 (2164): No heartbeat from core client for 30 sec - exiting 07:43:53 (2164): No heartbeat from core client for 30 sec - exiting 07:43:54 (2164): No heartbeat from core client for 30 sec - exiting 07:43:55 (2164): No heartbeat from core client for 30 sec - exiting 07:43:56 (2164): No heartbeat from core client for 30 sec - exiting 07:43:57 (2164): No heartbeat from core client for 30 sec - exiting 07:43:58 (2164): No heartbeat from core client for 30 sec - exiting 07:43:59 (2164): No heartbeat from core client for 30 sec - exiting 07:44:00 (2164): No heartbeat from core client for 30 sec - exiting 07:44:01 (2164): No heartbeat from core client for 30 sec - exiting 07:44:03 (2164): No heartbeat from core client for 30 sec - exiting 07:44:04 (2164): No heartbeat from core client for 30 sec - exiting 07:44:05 (2164): No heartbeat from core client for 30 sec - exiting 07:44:06 (2164): No heartbeat from core client for 30 sec - exiting 07:44:07 (2164): No heartbeat from core client for 30 sec - exiting 07:44:08 (2164): No heartbeat from core client for 30 sec - exiting 07:44:09 (2164): No heartbeat from core client for 30 sec - exiting 07:44:10 (2164): No heartbeat from core client for 30 sec - exiting 07:44:11 (2164): No heartbeat from core client for 30 sec - exiting 07:44:12 (2164): No heartbeat from core client for 30 sec - exiting 07:44:13 (2164): No heartbeat from core client for 30 sec - exiting 07:44:14 (2164): No heartbeat from core client for 30 sec - exiting 07:44:15 (2164): No heartbeat from core client for 30 sec - exiting 07:44:16 (2164): No heartbeat from core client for 30 sec - exiting 07:44:17 (2164): No heartbeat from core client for 30 sec - exiting 07:44:18 (2164): No heartbeat from core client for 30 sec - exiting 07:44:19 (2164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:44:20 (2164): No heartbeat from core client for 30 sec - exiting 07:44:21 (2164): No heartbeat from core client for 30 sec - exiting 07:44:22 (2164): No heartbeat from core client for 30 sec - exiting 07:44:24 (2164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 17:38:30 (8028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:00:31 (6236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:00:32 (6236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 18:16:59 (7524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:17:00 (7524): No heartbeat from core client for 30 sec - exiting 18:17:01 (7524): No heartbeat from core client for 30 sec - exiting 18:17:02 (7524): No heartbeat from core client for 30 sec - exiting 18:17:03 (7524): No heartbeat from core client for 30 sec - exiting 18:17:04 (7524): No heartbeat from core client for 30 sec - exiting 18:17:05 (7524): No heartbeat from core client for 30 sec - exiting 18:17:06 (7524): No heartbeat from core client for 30 sec - exiting 18:17:07 (7524): No heartbeat from core client for 30 sec - exiting 18:17:08 (7524): No heartbeat from core client for 30 sec - exiting 18:17:09 (7524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 21:00:51 (7328): No heartbeat from core client for 30 sec - exiting 21:00:52 (7328): No heartbeat from core client for 30 sec - exiting 21:00:53 (7328): No heartbeat from core client for 30 sec - exiting 21:00:54 (7328): No heartbeat from core client for 30 sec - exiting 21:00:55 (7328): No heartbeat from core client for 30 sec - exiting 21:00:56 (7328): No heartbeat from core client for 30 sec - exiting 21:00:57 (7328): No heartbeat from core client for 30 sec - exiting 21:00:58 (7328): No heartbeat from core client for 30 sec - exiting 21:00:59 (7328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:00 (7328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 17:57:36 (12292): No heartbeat from core client for 30 sec - exiting 17:57:38 (12292): No heartbeat from core client for 30 sec - exiting 17:57:39 (12292): No heartbeat from core client for 30 sec - exiting 17:57:40 (12292): No heartbeat from core client for 30 sec - exiting 17:57:41 (12292): No heartbeat from core client for 30 sec - exiting 17:57:42 (12292): No heartbeat from core client for 30 sec - exiting 17:57:43 (12292): No heartbeat from core client for 30 sec - exiting 17:57:44 (12292): No heartbeat from core client for 30 sec - exiting 17:57:45 (12292): No heartbeat from core client for 30 sec - exiting 17:57:46 (12292): No heartbeat from core client for 30 sec - exiting 17:57:47 (12292): No heartbeat from core client for 30 sec - exiting 17:57:48 (12292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9924, iMonCtr=1 Model crash detected, will try to restart... 14:30:10 (1724): No heartbeat from core client for 30 sec - exiting 14:30:11 (1724): No heartbeat from core client for 30 sec - exiting 14:30:12 (1724): No heartbeat from core client for 30 sec - exiting 14:30:13 (1724): No heartbeat from core client for 30 sec - exiting 14:30:14 (1724): No heartbeat from core client for 30 sec - exiting 14:30:15 (1724): No heartbeat from core client for 30 sec - exiting 14:30:16 (1724): No heartbeat from core client for 30 sec - exiting 14:30:17 (1724): No heartbeat from core client for 30 sec - exiting 14:30:18 (1724): No heartbeat from core client for 30 sec - exiting 14:30:19 (1724): No heartbeat from core client for 30 sec - exiting 14:30:20 (1724): No heartbeat from core client for 30 sec - exiting 14:30:21 (1724): No heartbeat from core client for 30 sec - exiting 14:30:22 (1724): No heartbeat from core client for 30 sec - exiting 14:30:23 (1724): No heartbeat from core client for 30 sec - exiting 14:30:24 (1724): No heartbeat from core client for 30 sec - exiting 14:30:25 (1724): No heartbeat from core client for 30 sec - exiting 14:30:26 (1724): No heartbeat from core client for 30 sec - exiting 14:30:27 (1724): No heartbeat from core client for 30 sec - exiting 14:30:28 (1724): No heartbeat from core client for 30 sec - exiting 14:30:29 (1724): No heartbeat from core client for 30 sec - exiting 14:30:30 (1724): No heartbeat from core client for 30 sec - exiting 14:30:31 (1724): No heartbeat from core client for 30 sec - exiting 14:30:32 (1724): No heartbeat from core client for 30 sec - exiting 14:30:33 (1724): No heartbeat from core client for 30 sec - exiting 14:30:34 (1724): No heartbeat from core client for 30 sec - exiting 14:30:35 (1724): No heartbeat from core client for 30 sec - exiting 14:30:36 (1724): No heartbeat from core client for 30 sec - exiting 14:30:37 (1724): No heartbeat from core client for 30 sec - exiting 14:30:38 (1724): No heartbeat from core client for 30 sec - exiting 14:30:39 (1724): No heartbeat from core client for 30 sec - exiting 14:30:41 (1724): No heartbeat from core client for 30 sec - exiting 14:30:42 (1724): No heartbeat from core client for 30 sec - exiting 14:30:43 (1724): No heartbeat from core client for 30 sec - exiting 14:30:44 (1724): No heartbeat from core client for 30 sec - exiting 14:30:45 (1724): No heartbeat from core client for 30 sec - exiting 14:30:46 (1724): No heartbeat from core client for 30 sec - exiting 14:30:47 (1724): No heartbeat from core client for 30 sec - exiting 14:30:48 (1724): No heartbeat from core client for 30 sec - exiting 14:30:49 (1724): No heartbeat from core client for 30 sec - exiting 14:30:50 (1724): No heartbeat from core client for 30 sec - exiting 14:30:51 (1724): No heartbeat from core client for 30 sec - exiting 14:30:53 (1724): No heartbeat from core client for 30 sec - exiting 14:30:54 (1724): No heartbeat from core client for 30 sec - exiting 14:30:55 (1724): No heartbeat from core client for 30 sec - exiting 14:30:56 (1724): No heartbeat from core client for 30 sec - exiting 14:30:57 (1724): No heartbeat from core client for 30 sec - exiting 14:30:58 (1724): No heartbeat from core client for 30 sec - exiting 14:30:59 (1724): No heartbeat from core client for 30 sec - exiting 14:31:00 (1724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:16:54 (3188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:23:40 (7936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:41 (7936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 20:51:28 (7648): No heartbeat from core client for 30 sec - exiting 20:51:29 (7648): No heartbeat from core client for 30 sec - exiting 20:51:31 (7648): No heartbeat from core client for 30 sec - exiting 20:51:32 (7648): No heartbeat from core client for 30 sec - exiting 20:51:33 (7648): No heartbeat from core client for 30 sec - exiting 20:51:34 (7648): No heartbeat from core client for 30 sec - exiting 20:51:35 (7648): No heartbeat from core client for 30 sec - exiting 20:51:36 (7648): No heartbeat from core client for 30 sec - exiting 20:51:37 (7648): No heartbeat from core client for 30 sec - exiting 20:51:38 (7648): No heartbeat from core client for 30 sec - exiting 20:51:39 (7648): No heartbeat from core client for 30 sec - exiting 20:51:40 (7648): No heartbeat from core client for 30 sec - exiting 20:51:41 (7648): No heartbeat from core client for 30 sec - exiting 20:51:43 (7648): No heartbeat from core client for 30 sec - exiting 20:51:44 (7648): No heartbeat from core client for 30 sec - exiting 20:51:45 (7648): No heartbeat from core client for 30 sec - exiting 20:51:46 (7648): No heartbeat from core client for 30 sec - exiting 20:51:47 (7648): No heartbeat from core client for 30 sec - exiting 20:51:48 (7648): No heartbeat from core client for 30 sec - exiting 20:51:49 (7648): No heartbeat from core client for 30 sec - exiting 20:51:50 (7648): No heartbeat from core client for 30 sec - exiting 20:51:51 (7648): No heartbeat from core client for 30 sec - exiting 20:51:52 (7648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:54:04 (6576): No heartbeat from core client for 30 sec - exiting 13:54:05 (6576): No heartbeat from core client for 30 sec - exiting 13:54:06 (6576): No heartbeat from core client for 30 sec - exiting 13:54:07 (6576): No heartbeat from core client for 30 sec - exiting 13:54:08 (6576): No heartbeat from core client for 30 sec - exiting 13:54:09 (6576): No heartbeat from core client for 30 sec - exiting 13:54:10 (6576): No heartbeat from core client for 30 sec - exiting 13:54:11 (6576): No heartbeat from core client for 30 sec - exiting 13:54:12 (6576): No heartbeat from core client for 30 sec - exiting 13:54:13 (6576): No heartbeat from core client for 30 sec - exiting 13:54:15 (6576): No heartbeat from core client for 30 sec - exiting 13:54:16 (6576): No heartbeat from core client for 30 sec - exiting 13:54:17 (6576): No heartbeat from core client for 30 sec - exiting 13:54:18 (6576): No heartbeat from core client for 30 sec - exiting 13:54:19 (6576): No heartbeat from core client for 30 sec - exiting 13:54:20 (6576): No heartbeat from core client for 30 sec - exiting 13:54:21 (6576): No heartbeat from core client for 30 sec - exiting 13:54:22 (6576): No heartbeat from core client for 30 sec - exiting 13:54:23 (6576): No heartbeat from core client for 30 sec - exiting 13:54:24 (6576): No heartbeat from core client for 30 sec - exiting 13:54:25 (6576): No heartbeat from core client for 30 sec - exiting 13:54:27 (6576): No heartbeat from core client for 30 sec - exiting 13:54:28 (6576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:13:07 (3900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:13:08 (3900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:03:18 (6740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:12:46 (12972): Can't acquire lockfile (32) - waiting 35s Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77E57353 read attempt to address 0x40335D95 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77383A93 read attempt to address 0x40335D9D Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o0fo_1940_40_007264632/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 18:01:40	1036099	12953630	hadcm3n_o0fo_1940_40_007264632_2	259,200	473,488	1.8267
25 Jul 2011 15:37:42	1036099	12953630	hadcm3n_o0fo_1940_40_007264632_2	233,280	422,234	1.8100
25 Jul 2011 12:57:04	1036099	12953630	hadcm3n_o0fo_1940_40_007264632_2	207,360	370,921	1.7888
10 Jul 2011 15:24:24	1036099	12953630	hadcm3n_o0fo_1940_40_007264632_2	181,440	320,275	1.7652
07 Jul 2011 19:33:09	1036099	12953630	hadcm3n_o0fo_1940_40_007264632_2	155,520	268,534	1.7267
04 Jul 2011 17:14:48	1036099	12953630	hadcm3n_o0fo_1940_40_007264632_2	129,600	216,135	1.6677
26 Jun 2011 12:10:45	1036099	12953630	hadcm3n_o0fo_1940_40_007264632_2	103,680	165,778	1.5989
10 Jun 2011 04:50:24	1036099	12953630	hadcm3n_o0fo_1940_40_007264632_2	77,760	122,757	1.5787
09 Jun 2011 16:12:59	1036099	12953630	hadcm3n_o0fo_1940_40_007264632_2	51,840	81,577	1.5736
08 Jun 2011 13:54:26	1036099	12953630	hadcm3n_o0fo_1940_40_007264632_2	25,920	42,067	1.6230