Task 14043346

Name	hadcm3n_t76h_1940_40_007743794_2
Workunit	7898903
Created	1 Feb 2012, 17:54:42 UTC
Sent	1 Feb 2012, 17:55:24 UTC
Report deadline	3 May 2012, 1:22:35 UTC
Received	9 Mar 2012, 9:04:25 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	932254
Run time	12 days 14 hours 26 min 28 sec
CPU time	9 days 21 hours 37 min 4 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.88 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 09:46:12 (5684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:31:10 (5412): No heartbeat from core client for 30 sec - exiting 11:31:11 (5412): No heartbeat from core client for 30 sec - exiting 11:31:12 (5412): No heartbeat from core client for 30 sec - exiting 11:31:13 (5412): No heartbeat from core client for 30 sec - exiting 11:31:14 (5412): No heartbeat from core client for 30 sec - exiting 11:31:15 (5412): No heartbeat from core client for 30 sec - exiting 11:31:16 (5412): No heartbeat from core client for 30 sec - exiting 11:31:17 (5412): No heartbeat from core client for 30 sec - exiting 11:31:18 (5412): No heartbeat from core client for 30 sec - exiting 11:31:19 (5412): No heartbeat from core client for 30 sec - exiting 11:31:20 (5412): No heartbeat from core client for 30 sec - exiting 11:31:21 (5412): No heartbeat from core client for 30 sec - exiting 11:31:22 (5412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:31:23 (5412): No heartbeat from core client for 30 sec - exiting 11:31:24 (5412): No heartbeat from core client for 30 sec - exiting 11:31:25 (5412): No heartbeat from core client for 30 sec - exiting 11:31:26 (5412): No heartbeat from core client for 30 sec - exiting 11:31:27 (5412): No heartbeat from core client for 30 sec - exiting 11:31:28 (5412): No heartbeat from core client for 30 sec - exiting 11:31:29 (5412): No heartbeat from core client for 30 sec - exiting 11:31:30 (5412): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 11:36:00 (5736): No heartbeat from core client for 30 sec - exiting 11:36:01 (5736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:42:59 (5392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:43:00 (5392): No heartbeat from core client for 30 sec - exiting 11:43:01 (5392): No heartbeat from core client for 30 sec - exiting 11:43:02 (5392): No heartbeat from core client for 30 sec - exiting 11:43:03 (5392): No heartbeat from core client for 30 sec - exiting 11:43:04 (5392): No heartbeat from core client for 30 sec - exiting 11:43:05 (5392): No heartbeat from core client for 30 sec - exiting 11:43:06 (5392): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:12:29 (5920): No heartbeat from core client for 30 sec - exiting 11:12:30 (5920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 11:09:13 (4892): No heartbeat from core client for 30 sec - exiting 11:09:14 (4892): No heartbeat from core client for 30 sec - exiting 11:09:16 (4892): No heartbeat from core client for 30 sec - exiting 11:09:17 (4892): No heartbeat from core client for 30 sec - exiting 11:09:18 (4892): No heartbeat from core client for 30 sec - exiting 11:09:19 (4892): No heartbeat from core client for 30 sec - exiting 11:09:20 (4892): No heartbeat from core client for 30 sec - exiting 11:09:21 (4892): No heartbeat from core client for 30 sec - exiting 11:09:22 (4892): No heartbeat from core client for 30 sec - exiting 11:09:23 (4892): No heartbeat from core client for 30 sec - exiting 11:09:24 (4892): No heartbeat from core client for 30 sec - exiting 11:09:25 (4892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=224, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 09:42:29 (4828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 12:10:47 (1716): No heartbeat from core client for 30 sec - exiting 12:10:48 (1716): No heartbeat from core client for 30 sec - exiting 12:10:49 (1716): No heartbeat from core client for 30 sec - exiting 12:10:50 (1716): No heartbeat from core client for 30 sec - exiting 12:10:51 (1716): No heartbeat from core client for 30 sec - exiting 12:10:52 (1716): No heartbeat from core client for 30 sec - exiting 12:10:53 (1716): No heartbeat from core client for 30 sec - exiting 12:10:54 (1716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 11:29:47 (5464): No heartbeat from core client for 30 sec - exiting 11:29:49 (5464): No heartbeat from core client for 30 sec - exiting 11:29:50 (5464): No heartbeat from core client for 30 sec - exiting 11:29:51 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:29:52 (5464): No heartbeat from core client for 30 sec - exiting 11:29:53 (5464): No heartbeat from core client for 30 sec - exiting 11:29:54 (5464): No heartbeat from core client for 30 sec - exiting 11:29:55 (5464): No heartbeat from core client for 30 sec - exiting 11:29:56 (5464): No heartbeat from core client for 30 sec - exiting 11:29:57 (5464): No heartbeat from core client for 30 sec - exiting 11:29:58 (5464): No heartbeat from core client for 30 sec - exiting 11:29:59 (5464): No heartbeat from core client for 30 sec - exiting 11:30:00 (5464): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:50:32 (1692): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:07:12 (4708): No heartbeat from core client for 30 sec - exiting 10:07:13 (4708): No heartbeat from core client for 30 sec - exiting 10:07:14 (4708): No heartbeat from core client for 30 sec - exiting 10:07:15 (4708): No heartbeat from core client for 30 sec - exiting 10:07:16 (4708): No heartbeat from core client for 30 sec - exiting 10:07:17 (4708): No heartbeat from core client for 30 sec - exiting 10:07:18 (4708): No heartbeat from core client for 30 sec - exiting 10:07:19 (4708): No heartbeat from core client for 30 sec - exiting 10:07:20 (4708): No heartbeat from core client for 30 sec - exiting 10:07:21 (4708): No heartbeat from core client for 30 sec - exiting 10:07:22 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:23 (4708): No heartbeat from core client for 30 sec - exiting 10:07:24 (4708): No heartbeat from core client for 30 sec - exiting 10:07:25 (4708): No heartbeat from core client for 30 sec - exiting 10:07:26 (4708): No heartbeat from core client for 30 sec - exiting 10:07:27 (4708): No heartbeat from core client for 30 sec - exiting 10:07:28 (4708): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:59:50 (2264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:51 (2264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 10:48:11 (4936): No heartbeat from core client for 30 sec - exiting 10:48:12 (4936): No heartbeat from core client for 30 sec - exiting 10:48:14 (4936): No heartbeat from core client for 30 sec - exiting 10:48:15 (4936): No heartbeat from core client for 30 sec - exiting 10:48:16 (4936): No heartbeat from core client for 30 sec - exiting 10:48:17 (4936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:18 (4936): No heartbeat from core client for 30 sec - exiting 10:48:20 (4936): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:05:00 (2816): No heartbeat from core client for 30 sec - exiting 10:05:01 (2816): No heartbeat from core client for 30 sec - exiting 10:05:02 (2816): No heartbeat from core client for 30 sec - exiting 10:05:03 (2816): No heartbeat from core client for 30 sec - exiting 10:05:04 (2816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:05:05 (2816): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:17:54 (4920): No heartbeat from core client for 30 sec - exiting 10:17:55 (4920): No heartbeat from core client for 30 sec - exiting 10:17:56 (4920): No heartbeat from core client for 30 sec - exiting 10:17:57 (4920): No heartbeat from core client for 30 sec - exiting 10:17:58 (4920): No heartbeat from core client for 30 sec - exiting 10:17:59 (4920): No heartbeat from core client for 30 sec - exiting 10:18:00 (4920): No heartbeat from core client for 30 sec - exiting 10:18:01 (4920): No heartbeat from core client for 30 sec - exiting 10:18:02 (4920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:18:03 (4920): No heartbeat from core client for 30 sec - exiting 10:18:04 (4920): No heartbeat from core client for 30 sec - exiting 10:18:05 (4920): No heartbeat from core client for 30 sec - exiting 10:18:06 (4920): No heartbeat from core client for 30 sec - exiting 10:18:07 (4920): No heartbeat from core client for 30 sec - exiting 10:18:08 (4920): No heartbeat from core client for 30 sec - exiting 10:18:09 (4920): No heartbeat from core client for 30 sec - exiting 10:18:10 (4920): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:54:52 (4064): No heartbeat from core client for 30 sec - exiting 10:54:54 (4064): No heartbeat from core client for 30 sec - exiting 10:54:55 (4064): No heartbeat from core client for 30 sec - exiting 10:54:56 (4064): No heartbeat from core client for 30 sec - exiting 10:54:57 (4064): No heartbeat from core client for 30 sec - exiting 10:54:58 (4064): No heartbeat from core client for 30 sec - exiting 10:54:59 (4064): No heartbeat from core client for 30 sec - exiting 10:55:00 (4064): No heartbeat from core client for 30 sec - exiting 10:55:01 (4064): No heartbeat from core client for 30 sec - exiting 10:55:02 (4064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:55:03 (4064): No heartbeat from core client for 30 sec - exiting 10:55:04 (4064): No heartbeat from core client for 30 sec - exiting 10:55:05 (4064): No heartbeat from core client for 30 sec - exiting 10:55:06 (4064): No heartbeat from core client for 30 sec - exiting 10:55:07 (4064): No heartbeat from core client for 30 sec - exiting 10:55:08 (4064): No heartbeat from core client for 30 sec - exiting 10:55:09 (4064): No heartbeat from core client for 30 sec - exiting 10:55:10 (4064): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:16:53 (4380): No heartbeat from core client for 30 sec - exiting 10:16:54 (4380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910CCE read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6000, selfPID=6000, iMonCtr=1 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x400A0755 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Mar 2012 22:48:47	932254	14043346	hadcm3n_t76h_1940_40_007743794_2	259,200	782,443	3.0187
05 Mar 2012 23:35:45	932254	14043346	hadcm3n_t76h_1940_40_007743794_2	233,280	693,778	2.9740
04 Mar 2012 11:00:56	932254	14043346	hadcm3n_t76h_1940_40_007743794_2	207,360	611,468	2.9488
02 Mar 2012 12:54:13	932254	14043346	hadcm3n_t76h_1940_40_007743794_2	181,440	529,509	2.9184
29 Feb 2012 10:53:19	932254	14043346	hadcm3n_t76h_1940_40_007743794_2	155,520	448,606	2.8846
12 Feb 2012 11:06:12	932254	14043346	hadcm3n_t76h_1940_40_007743794_2	129,600	368,164	2.8408
09 Feb 2012 19:36:07	932254	14043346	hadcm3n_t76h_1940_40_007743794_2	103,680	311,581	3.0052
07 Feb 2012 11:54:36	932254	14043346	hadcm3n_t76h_1940_40_007743794_2	77,760	233,244	2.9995
05 Feb 2012 09:40:30	932254	14043346	hadcm3n_t76h_1940_40_007743794_2	51,840	155,671	3.0029
03 Feb 2012 13:44:17	932254	14043346	hadcm3n_t76h_1940_40_007743794_2	25,920	77,470	2.9888