Task 13406914

Name	hadcm3n_u377_1980_40_007458462_1
Workunit	7655965
Created	22 Sep 2011, 14:53:14 UTC
Sent	22 Sep 2011, 14:54:05 UTC
Report deadline	22 Dec 2011, 22:21:16 UTC
Received	10 Oct 2011, 21:45:26 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1149416
Run time	7 hours 52 min 32 sec
CPU time
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.70 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> 12:47:56 (3832): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 17:19:03 (4756): start_timer_thread(): CreateThread() failed, errno 0 17:28:55 (4416): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... 19:10:51 (4944): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 19:12:28 (6772): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:42:48 (4620): start_timer_thread(): CreateThread() failed, errno 0 22:07:59 (5100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:08:00 (5100): No heartbeat from core client for 30 sec - exiting 22:08:02 (5100): No heartbeat from core client for 30 sec - exiting 22:09:17 (6720): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:46:17 (6220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:46:18 (6220): No heartbeat from core client for 30 sec - exiting 00:46:20 (6220): No heartbeat from core client for 30 sec - exiting 00:46:21 (6220): No heartbeat from core client for 30 sec - exiting 00:46:22 (6220): No heartbeat from core client for 30 sec - exiting 00:48:07 (6204): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:13:23 (2188): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 01:13:25 (2188): No heartbeat from core client for 30 sec - exiting 02:42:30 (896): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 09:30:07 (1384): start_timer_thread(): CreateThread() failed, errno 0 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 09:31:37 (4908): start_timer_thread(): CreateThread() failed, errno 0 09:31:55 (4468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:55:03 (5828): No heartbeat from core client for 30 sec - exiting 01:55:04 (5828): No heartbeat from core client for 30 sec - exiting 01:55:05 (5828): No heartbeat from core client for 30 sec - exiting 01:55:06 (5828): No heartbeat from core client for 30 sec - exiting 01:55:07 (5828): No heartbeat from core client for 30 sec - exiting 01:55:08 (5828): No heartbeat from core client for 30 sec - exiting 01:55:09 (5828): No heartbeat from core client for 30 sec - exiting 01:55:10 (5828): No heartbeat from core client for 30 sec - exiting 01:55:11 (5828): No heartbeat from core client for 30 sec - exiting 01:55:12 (5828): No heartbeat from core client for 30 sec - exiting 01:55:13 (5828): No heartbeat from core client for 30 sec - exiting 01:55:14 (5828): No heartbeat from core client for 30 sec - exiting 01:55:15 (5828): No heartbeat from core client for 30 sec - exiting 01:55:16 (5828): No heartbeat from core client for 30 sec - exiting 01:55:17 (5828): No heartbeat from core client for 30 sec - exiting 01:55:18 (5828): No heartbeat from core client for 30 sec - exiting 01:55:19 (5828): No heartbeat from core client for 30 sec - exiting 01:55:20 (5828): No heartbeat from core client for 30 sec - exiting 01:55:21 (5828): No heartbeat from core client for 30 sec - exiting 01:55:22 (5828): No heartbeat from core client for 30 sec - exiting 01:55:23 (5828): No heartbeat from core client for 30 sec - exiting 01:55:24 (5828): No heartbeat from core client for 30 sec - exiting 01:55:25 (5828): No heartbeat from core client for 30 sec - exiting 01:55:26 (5828): No heartbeat from core client for 30 sec - exiting 01:55:27 (5828): No heartbeat from core client for 30 sec - exiting 01:55:28 (5828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:40:55 (1496): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 22:13:47 (4756): start_timer_thread(): CreateThread() failed, errno 0 22:27:00 (6092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:03 (6092): No heartbeat from core client for 30 sec - exiting 22:28:48 (6832): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 00:33:11 (7696): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 16:54:27 (7860): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 18:32:39 (1492): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 23:55:09 (5548): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:46:28 (6508): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:09:57 (5376): No heartbeat from core client for 30 sec - exiting 19:09:58 (5376): No heartbeat from core client for 30 sec - exiting 19:09:59 (5376): No heartbeat from core client for 30 sec - exiting 19:10:00 (5376): No heartbeat from core client for 30 sec - exiting 19:10:01 (5376): No heartbeat from core client for 30 sec - exiting 19:10:02 (5376): No heartbeat from core client for 30 sec - exiting 19:10:03 (5376): No heartbeat from core client for 30 sec - exiting 19:10:04 (5376): No heartbeat from core client for 30 sec - exiting 19:10:05 (5376): No heartbeat from core client for 30 sec - exiting 19:10:06 (5376): No heartbeat from core client for 30 sec - exiting 19:10:07 (5376): No heartbeat from core client for 30 sec - exiting 19:10:08 (5376): No heartbeat from core client for 30 sec - exiting 19:10:09 (5376): No heartbeat from core client for 30 sec - exiting 19:10:10 (5376): No heartbeat from core client for 30 sec - exiting 19:10:11 (5376): No heartbeat from core client for 30 sec - exiting 19:10:12 (5376): No heartbeat from core client for 30 sec - exiting 19:10:13 (5376): No heartbeat from core client for 30 sec - exiting 19:10:14 (5376): No heartbeat from core client for 30 sec - exiting 19:10:15 (5376): No heartbeat from core client for 30 sec - exiting 19:10:16 (5376): No heartbeat from core client for 30 sec - exiting 19:10:17 (5376): No heartbeat from core client for 30 sec - exiting 19:10:18 (5376): No heartbeat from core client for 30 sec - exiting 19:10:19 (5376): No heartbeat from core client for 30 sec - exiting 19:10:20 (5376): No heartbeat from core client for 30 sec - exiting 19:10:21 (5376): No heartbeat from core client for 30 sec - exiting 19:10:22 (5376): No heartbeat from core client for 30 sec - exiting 19:10:23 (5376): No heartbeat from core client for 30 sec - exiting 19:10:24 (5376): No heartbeat from core client for 30 sec - exiting 19:10:25 (5376): No heartbeat from core client for 30 sec - exiting 19:10:26 (5376): No heartbeat from core client for 30 sec - exiting 19:10:27 (5376): No heartbeat from core client for 30 sec - exiting 19:10:28 (5376): No heartbeat from core client for 30 sec - exiting 19:10:29 (5376): No heartbeat from core client for 30 sec - exiting 19:10:30 (5376): No heartbeat from core client for 30 sec - exiting 19:10:31 (5376): No heartbeat from core client for 30 sec - exiting 19:10:32 (5376): No heartbeat from core client for 30 sec - exiting 19:10:33 (5376): No heartbeat from core client for 30 sec - exiting 19:10:34 (5376): No heartbeat from core client for 30 sec - exiting 19:10:35 (5376): No heartbeat from core client for 30 sec - exiting 19:10:36 (5376): No heartbeat from core client for 30 sec - exiting 19:10:37 (5376): No heartbeat from core client for 30 sec - exiting 19:10:38 (5376): No heartbeat from core client for 30 sec - exiting 19:10:39 (5376): No heartbeat from core client for 30 sec - exiting 19:10:40 (5376): No heartbeat from core client for 30 sec - exiting 19:10:41 (5376): No heartbeat from core client for 30 sec - exiting 19:10:42 (5376): No heartbeat from core client for 30 sec - exiting 19:10:43 (5376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:10:44 (5376): No heartbeat from core client for 30 sec - exiting 19:11:00 (4956): start_timer_thread(): CreateThread() failed, errno 0 20:42:13 (3292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:42:38 (3292): No heartbeat from core client for 30 sec - exiting 20:47:54 (3536): No heartbeat from core client for 30 sec - exiting 20:47:55 (3536): No heartbeat from core client for 30 sec - exiting 20:47:56 (3536): No heartbeat from core client for 30 sec - exiting 20:47:57 (3536): No heartbeat from core client for 30 sec - exiting 20:47:58 (3536): No heartbeat from core client for 30 sec - exiting 20:47:59 (3536): No heartbeat from core client for 30 sec - exiting 20:48:00 (3536): No heartbeat from core client for 30 sec - exiting 20:48:01 (3536): No heartbeat from core client for 30 sec - exiting 20:48:02 (3536): No heartbeat from core client for 30 sec - exiting 20:48:03 (3536): No heartbeat from core client for 30 sec - exiting 20:48:34 (3536): No heartbeat from core client for 30 sec - exiting 20:48:35 (3536): No heartbeat from core client for 30 sec - exiting 20:48:36 (3536): No heartbeat from core client for 30 sec - exiting 20:48:37 (3536): No heartbeat from core client for 30 sec - exiting 20:48:38 (3088): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 20:49:06 (6348): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:06:30 (1100): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 04:29:51 (7344): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 15:14:05 (7516): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 03:26:10 (7308): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 15:53:23 (5188): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:24:55 (2564): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:29:13 (2156): start_timer_thread(): CreateThread() failed, errno 0 19:35:47 (3256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:36:44 (7512): start_timer_thread(): CreateThread() failed, errno 0 19:57:37 (7200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:57:38 (7200): No heartbeat from core client for 30 sec - exiting 16:02:06 (5272): No heartbeat from core client for 30 sec - exiting 16:02:07 (5272): No heartbeat from core client for 30 sec - exiting 16:02:08 (5272): No heartbeat from core client for 30 sec - exiting 16:02:09 (5272): No heartbeat from core client for 30 sec - exiting 16:02:10 (5272): No heartbeat from core client for 30 sec - exiting 16:02:11 (5272): No heartbeat from core client for 30 sec - exiting 16:02:12 (5272): No heartbeat from core client for 30 sec - exiting 16:02:13 (5272): No heartbeat from core client for 30 sec - exiting 16:02:14 (5272): No heartbeat from core client for 30 sec - exiting 16:02:15 (5272): No heartbeat from core client for 30 sec - exiting 16:02:16 (5272): No heartbeat from core client for 30 sec - exiting 16:02:17 (5272): No heartbeat from core client for 30 sec - exiting 16:02:18 (5272): No heartbeat from core client for 30 sec - exiting 16:02:19 (5272): No heartbeat from core client for 30 sec - exiting 16:02:20 (5272): No heartbeat from core client for 30 sec - exiting 16:02:21 (5272): No heartbeat from core client for 30 sec - exiting 16:02:22 (5272): No heartbeat from core client for 30 sec - exiting 16:02:23 (5272): No heartbeat from core client for 30 sec - exiting 16:02:24 (5272): No heartbeat from core client for 30 sec - exiting 16:02:25 (5272): No heartbeat from core client for 30 sec - exiting 16:02:26 (5272): No heartbeat from core client for 30 sec - exiting 16:02:27 (5272): No heartbeat from core client for 30 sec - exiting 16:02:28 (5272): No heartbeat from core client for 30 sec - exiting 16:02:29 (5272): No heartbeat from core client for 30 sec - exiting 16:02:30 (5272): No heartbeat from core client for 30 sec - exiting 16:02:31 (5272): No heartbeat from core client for 30 sec - exiting 16:02:32 (5272): No heartbeat from core client for 30 sec - exiting 16:02:33 (5272): No heartbeat from core client for 30 sec - exiting 16:02:34 (5272): No heartbeat from core client for 30 sec - exiting 16:02:35 (5272): No heartbeat from core client for 30 sec - exiting 16:02:36 (5272): No heartbeat from core client for 30 sec - exiting 16:02:37 (5272): No heartbeat from core client for 30 sec - exiting 16:02:38 (5272): No heartbeat from core client for 30 sec - exiting 16:02:39 (5272): No heartbeat from core client for 30 sec - exiting 16:02:40 (5272): No heartbeat from core client for 30 sec - exiting 16:02:41 (5272): No heartbeat from core client for 30 sec - exiting 16:02:42 (5272): No heartbeat from core client for 30 sec - exiting 16:02:44 (5272): No heartbeat from core client for 30 sec - exiting 16:02:45 (5272): No heartbeat from core client for 30 sec - exiting 16:02:46 (5272): No heartbeat from core client for 30 sec - exiting 16:02:47 (5272): No heartbeat from core client for 30 sec - exiting 16:02:48 (5272): No heartbeat from core client for 30 sec - exiting 16:02:49 (5272): No heartbeat from core client for 30 sec - exiting 16:02:50 (5272): No heartbeat from core client for 30 sec - exiting 16:02:51 (5272): No heartbeat from core client for 30 sec - exiting 16:02:52 (5272):CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting 16:02:53 (5272): No heartbeat from core client for 30 sec - exiting 16:03:56 (5520): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:37:31 (5624): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:13:14 (4112): start_timer_thread(): CreateThread() failed, errno 0 forrtl: The requested operation cannot be performed on a file with a user-mapped section open. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4852, iMonCtr=1 Model crash detected, will try to restart... 13:15:40 (5980): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 22:02:35 (5036): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:04:55 (5124): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 16:04:56 (5124): No heartbeat from core client for 30 sec - exiting 16:04:57 (5124): No heartbeat from core client for 30 sec - exiting 16:04:58 (5124): No heartbeat from core client for 30 sec - exiting 16:04:59 (5124): No heartbeat from core client for 30 sec - exiting 21:00:26 (7544): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:12:24 (5072): No heartbeat from core client for 30 sec - exiting 21:12:25 (5072): No heartbeat from core client for 30 sec - exiting 21:12:26 (5072): No heartbeat from core client for 30 sec - exiting 21:12:27 (5072): No heartbeat from core client for 30 sec - exiting 21:12:28 (5072): No heartbeat from core client for 30 sec - exiting 21:12:30 (5072): No heartbeat from core client for 30 sec - exiting 21:12:31 (5072): No heartbeat from core client for 30 sec - exiting 21:12:32 (5072): No heartbeat from core client for 30 sec - exiting 21:12:33 (5072): No heartbeat from core client for 30 sec - exiting 21:12:34 (5072): No heartbeat from core client for 30 sec - exiting 21:13:15 (5072):CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting 21:13:44 (5732): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C911780 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5732, selfPID=5732, iMonCtr=1 </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Oct 2011 21:18:20	1149416	13406914	hadcm3n_u377_1980_40_007458462_1	259,200	26,914	0.1038
08 Oct 2011 09:44:56	1149416	13406914	hadcm3n_u377_1980_40_007458462_1	233,280	25,452	0.1091
07 Oct 2011 02:08:05	1149416	13406914	hadcm3n_u377_1980_40_007458462_1	207,360	5,079	0.0245
05 Oct 2011 01:53:08	1149416	13406914	hadcm3n_u377_1980_40_007458462_1	181,440	3,156	0.0174
01 Oct 2011 05:30:36	1149416	13406914	hadcm3n_u377_1980_40_007458462_1	155,520	40,143	0.2581
28 Sep 2011 11:06:00	1149416	13406914	hadcm3n_u377_1980_40_007458462_1	129,600	24,737	0.1909
27 Sep 2011 10:54:32	1149416	13406914	hadcm3n_u377_1980_40_007458462_1	103,680	7,435	0.0717
25 Sep 2011 18:29:06	1149416	13406914	hadcm3n_u377_1980_40_007458462_1	77,760	40,600	0.5221
25 Sep 2011 01:00:30	1149416	13406914	hadcm3n_u377_1980_40_007458462_1	51,840	34,320	0.6620
23 Sep 2011 13:29:20	1149416	13406914	hadcm3n_u377_1980_40_007458462_1	25,920	47,951	1.8500