Task 13405378

Name	hadcm3n_p04i_1940_40_007419374_2
Workunit	7617009
Created	22 Sep 2011, 9:48:50 UTC
Sent	22 Sep 2011, 9:49:34 UTC
Report deadline	22 Dec 2011, 17:16:45 UTC
Received	26 Oct 2011, 5:08:45 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1091933
Run time	6 days 5 hours 40 min 2 sec
CPU time	5 days 16 hours 41 min 16 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.79 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> 07:06:43 (3696): No heartbeat from core client for 30 sec - exiting 07:06:44 (3696): No heartbeat from core client for 30 sec - exiting 07:06:45 (3696): No heartbeat from core client for 30 sec - exiting 07:06:46 (3696): No heartbeat from core client for 30 sec - exiting 07:06:47 (3696): No heartbeat from core client for 30 sec - exiting 07:06:48 (3696): No heartbeat from core client for 30 sec - exiting 07:06:49 (3696): No heartbeat from core client for 30 sec - exiting 07:06:50 (3696): No heartbeat from core client for 30 sec - exiting 07:06:51 (3696): No heartbeat from core client for 30 sec - exiting 07:06:52 (3696): No heartbeat from core client for 30 sec - exiting 07:06:54 (3696): No heartbeat from core client for 30 sec - exiting 07:06:55 (3696): No heartbeat from core client for 30 sec - exiting 07:06:56 (3696): No heartbeat from core client for 30 sec - exiting 07:06:57 (3696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3440, iMonCtr=1 Model crash detected, will try to restart... 07:13:27 (1000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:11:19 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:05:05 (1968): No heartbeat from core client for 30 sec - exiting 07:05:06 (1968): No heartbeat from core client for 30 sec - exiting 07:05:07 (1968): No heartbeat from core client for 30 sec - exiting 07:05:08 (1968): No heartbeat from core client for 30 sec - exiting 07:05:09 (1968): No heartbeat from core client for 30 sec - exiting 07:05:10 (1968): No heartbeat from core client for 30 sec - exiting 07:05:11 (1968): No heartbeat from core client for 30 sec - exiting 07:05:12 (1968): No heartbeat from core client for 30 sec - exiting 07:05:13 (1968): No heartbeat from core client for 30 sec - exiting 07:05:14 (1968): No heartbeat from core client for 30 sec - exiting 07:05:16 (1968): No heartbeat from core client for 30 sec - exiting 07:05:17 (1968): No heartbeat from core client for 30 sec - exiting 07:05:18 (1968): No heartbeat from core client for 30 sec - exiting 07:05:19 (1968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:05:20 (1968): No heartbeat from core client for 30 sec - exiting 07:05:21 (1968): No heartbeat from core client for 30 sec - exiting 07:05:22 (1968): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4372, iMonCtr=1 Model crash detected, will try to restart... 07:01:55 (2720): No heartbeat from core client for 30 sec - exiting 07:01:56 (2720): No heartbeat from core client for 30 sec - exiting 07:01:57 (2720): No heartbeat from core client for 30 sec - exiting 07:01:59 (2720): No heartbeat from core client for 30 sec - exiting 07:02:00 (2720): No heartbeat from core client for 30 sec - exiting 07:02:01 (2720): No heartbeat from core client for 30 sec - exiting 07:02:02 (2720): No heartbeat from core client for 30 sec - exiting 07:02:03 (2720): No heartbeat from core client for 30 sec - exiting 07:02:04 (2720): No heartbeat from core client for 30 sec - exiting 07:02:05 (2720): No heartbeat from core client for 30 sec - exiting 07:02:06 (2720): No heartbeat from core client for 30 sec - exiting 07:02:07 (2720): No heartbeat from core client for 30 sec - exiting 07:02:08 (2720): No heartbeat from core client for 30 sec - exiting 07:02:09 (2720): No heartbeat from core client for 30 sec - exiting 07:02:11 (2720): No heartbeat from core client for 30 sec - exiting 07:02:12 (2720): No heartbeat from core client for 30 sec - exiting 07:02:13 (2720): No heartbeat from core client for 30 sec - exiting 07:02:14 (2720): No heartbeat from core client for 30 sec - exiting 07:02:15 (2720): No heartbeat from core client for 30 sec - exiting 07:02:16 (2720): No heartbeat from core client for 30 sec - exiting 07:02:17 (2720): No heartbeat from core client for 30 sec - exiting 07:02:18 (2720): No heartbeat from core client for 30 sec - exiting 07:02:19 (2720): No heartbeat from core client for 30 sec - exiting 07:02:20 (2720): No heartbeat from core client for 30 sec - exiting 07:02:21 (2720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2436, iMonCtr=1 Model crash detected, will try to restart... 07:06:05 (3972): No heartbeat from core client for 30 sec - exiting 07:06:06 (3972): No heartbeat from core client for 30 sec - exiting 07:06:07 (3972): No heartbeat from core client for 30 sec - exiting 07:06:08 (3972): No heartbeat from core client for 30 sec - exiting 07:06:09 (3972): No heartbeat from core client for 30 sec - exiting 07:06:10 (3972): No heartbeat from core client for 30 sec - exiting 07:06:11 (3972): No heartbeat from core client for 30 sec - exiting 07:06:12 (3972): No heartbeat from core client for 30 sec - exiting 07:06:13 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:15 (3972): No heartbeat from core client for 30 sec - exiting 07:06:56 (880): No heartbeat from core client for 30 sec - exiting 07:06:58 (880): No heartbeat from core client for 30 sec - exiting 07:06:59 (880): No heartbeat from core client for 30 sec - exiting 07:07:00 (880): No heartbeat from core client for 30 sec - exiting 07:07:01 (880): No heartbeat from core client for 30 sec - exiting 07:07:02 (880): No heartbeat from core client for 30 sec - exiting 07:07:03 (880): No heartbeat from core client for 30 sec - exiting 07:07:04 (880): No heartbeat from core client for 30 sec - exiting 07:07:05 (880): No heartbeat from core client for 30 sec - exiting 07:07:06 (880): No heartbeat from core client for 30 sec - exiting 07:07:07 (880): No heartbeat from core client for 30 sec - exiting 07:07:09 (880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4056, iMonCtr=1 Model crash detected, will try to restart... 07:00:29 (3068): No heartbeat from core client for 30 sec - exiting 07:00:30 (3068): No heartbeat from core client for 30 sec - exiting 07:00:31 (3068): No heartbeat from core client for 30 sec - exiting 07:00:32 (3068): No heartbeat from core client for 30 sec - exiting 07:00:34 (3068): No heartbeat from core client for 30 sec - exiting 07:00:35 (3068): No heartbeat from core client for 30 sec - exiting 07:00:36 (3068): No heartbeat from core client for 30 sec - exiting 07:00:37 (3068): No heartbeat from core client for 30 sec - exiting 07:00:38 (3068): No heartbeat from core client for 30 sec - exiting 07:00:39 (3068): No heartbeat from core client for 30 sec - exiting 07:00:40 (3068): No heartbeat from core client for 30 sec - exiting 07:00:41 (3068): No heartbeat from core client for 30 sec - exiting 07:00:42 (3068): No heartbeat from core client for 30 sec - exiting 07:00:43 (3068): No heartbeat from core client for 30 sec - exiting 07:00:45 (3068): No heartbeat from core client for 30 sec - exiting 07:00:46 (3068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:00:47 (1416): Can't acquire lockfile (32) - waiting 35s Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1416, iMonCtr=1 Model crash detected, will try to restart... 14:18:42 (4108): No heartbeat from core client for 30 sec - exiting 14:18:44 (4108): No heartbeat from core client for 30 sec - exiting 14:18:45 (4108): No heartbeat from core client for 30 sec - exiting 14:18:46 (4108): No heartbeat from core client for 30 sec - exiting 14:18:47 (4108): No heartbeat from core client for 30 sec - exiting 14:18:48 (4108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:18:49 (4108): No heartbeat from core client for 30 sec - exiting 07:03:37 (3000): No heartbeat from core client for 30 sec - exiting 07:03:38 (3000): No heartbeat from core client for 30 sec - exiting 07:03:39 (3000): No heartbeat from core client for 30 sec - exiting 07:03:40 (3000): No heartbeat from core client for 30 sec - exiting 07:03:41 (3000): No heartbeat from core client for 30 sec - exiting 07:03:42 (3000): No heartbeat from core client for 30 sec - exiting 07:03:43 (3000): No heartbeat from core client for 30 sec - exiting 07:03:44 (3000): No heartbeat from core client for 30 sec - exiting 07:03:45 (3000): No heartbeat from core client for 30 sec - exiting 07:03:46 (3000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1212, iMonCtr=1 Model crash detected, will try to restart... 07:07:18 (3000): No heartbeat from core client for 30 sec - exiting 07:07:19 (3000): No heartbeat from core client for 30 sec - exiting 07:07:20 (3000): No heartbeat from core client for 30 sec - exiting 07:07:22 (3000): No heartbeat from core client for 30 sec - exiting 07:07:23 (3000): No heartbeat from core client for 30 sec - exiting 07:07:24 (3000): No heartbeat from core client for 30 sec - exiting 07:07:25 (3000): No heartbeat from core client for 30 sec - exiting 07:07:26 (3000): No heartbeat from core client for 30 sec - exiting 07:07:27 (3000): No heartbeat from core client for 30 sec - exiting 07:07:28 (3000): No heartbeat from core client for 30 sec - exiting 07:07:29 (3000): No heartbeat from core client for 30 sec - exiting 07:07:30 (3000): No heartbeat from core client for 30 sec - exiting 07:07:31 (3000): No heartbeat from core client for 30 sec - exiting 07:07:32 (3000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3172, iMonCtr=1 Model crash detected, will try to restart... 07:06:52 (1948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4396, iMonCtr=1 Model crash detected, will try to restart... 07:20:20 (2060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:41 (1072): No heartbeat from core client for 30 sec - exiting 07:31:42 (1072): No heartbeat from core client for 30 sec - exiting 07:31:43 (1072): No heartbeat from core client for 30 sec - exiting 07:31:44 (1072): No heartbeat from core client for 30 sec - exiting 07:31:45 (1072): No heartbeat from core client for 30 sec - exiting 07:31:46 (1072): No heartbeat from core client for 30 sec - exiting 07:31:47 (1072): No heartbeat from core client for 30 sec - exiting 07:31:48 (1072): No heartbeat from core client for 30 sec - exiting 07:31:49 (1072): No heartbeat from core client for 30 sec - exiting 07:31:51 (1072): No heartbeat from core client for 30 sec - exiting 07:31:52 (1072): No heartbeat from core client for 30 sec - exiting 07:31:53 (1072): No heartbeat from core client for 30 sec - exiting 07:31:54 (1072): No heartbeat from core client for 30 sec - exiting 07:31:55 (1072): No heartbeat from core client for 30 sec - exiting 07:31:56 (1072): No heartbeat from core client for 30 sec - exiting 07:31:57 (1072): No heartbeat from core client for 30 sec - exiting 07:31:58 (1072): No heartbeat from core client for 30 sec - exiting 07:31:59 (1072): No heartbeat from core client for 30 sec - exiting 07:32:00 (1072): No heartbeat from core client for 30 sec - exiting 07:32:01 (1072): No heartbeat from core client for 30 sec - exiting 07:32:03 (1072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:04:15 (2104): No heartbeat from core client for 30 sec - exiting 07:04:16 (2104): No heartbeat from core client for 30 sec - exiting 07:04:17 (2104): No heartbeat from core client for 30 sec - exiting 07:04:18 (2104): No heartbeat from core client for 30 sec - exiting 07:04:19 (2104): No heartbeat from core client for 30 sec - exiting 07:04:20 (2104): No heartbeat from core client for 30 sec - exiting 07:04:22 (2104): No heartbeat from core client for 30 sec - exiting 07:04:23 (2104): No heartbeat from core client for 30 sec - exiting 07:04:24 (2104): No heartbeat from core client for 30 sec - exiting 07:04:25 (2104): No heartbeat from core client for 30 sec - exiting 07:04:26 (2104): No heartbeat from core client for 30 sec - exiting 07:04:27 (2104): No heartbeat from core client for 30 sec - exiting 07:04:28 (2104): No heartbeat from core client for 30 sec - exiting 07:04:29 (2104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4228, iMonCtr=1 Model crash detected, will try to restart... 07:03:39 (3420): No heartbeat from core client for 30 sec - exiting 07:03:41 (3420): No heartbeat from core client for 30 sec - exiting 07:03:42 (3420): No heartbeat from core client for 30 sec - exiting 07:03:43 (3420): No heartbeat from core client for 30 sec - exiting 07:03:44 (3420): No heartbeat from core client for 30 sec - exiting 07:03:45 (3420): No heartbeat from core client for 30 sec - exiting 07:03:46 (3420): No heartbeat from core client for 30 sec - exiting 07:03:47 (3420): No heartbeat from core client for 30 sec - exiting 07:03:48 (3420): No heartbeat from core client for 30 sec - exiting 07:03:49 (3420): No heartbeat from core client for 30 sec - exiting 07:03:50 (3420): No heartbeat from core client for 30 sec - exiting 07:03:52 (3420): No heartbeat from core client for 30 sec - exiting 07:03:53 (3420): No heartbeat from core client for 30 sec - exiting 07:03:54 (3420): No heartbeat from core client for 30 sec - exiting 07:03:55 (3420): No heartbeat from core client for 30 sec - exiting 07:03:56 (3420): No heartbeat from core client for 30 sec - exiting 07:03:57 (3420): No heartbeat from core client for 30 sec - exiting 07:03:58 (3420): No heartbeat from core client for 30 sec - exiting 07:03:59 (3420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:28 (2640): No heartbeat from core client for 30 sec - exiting 07:03:29 (2640): No heartbeat from core client for 30 sec - exiting 07:03:30 (2640): No heartbeat from core client for 30 sec - exiting 07:03:31 (2640): No heartbeat from core client for 30 sec - exiting 07:03:32 (2640): No heartbeat from core client for 30 sec - exiting 07:03:33 (2640): No heartbeat from core client for 30 sec - exiting 07:03:34 (2640): No heartbeat from core client for 30 sec - exiting 07:03:35 (2640): No heartbeat from core client for 30 sec - exiting 07:03:37 (2640): No heartbeat from core client for 30 sec - exiting 07:03:38 (2640): No heartbeat from core client for 30 sec - exiting 07:03:39 (2640): No heartbeat from core client for 30 sec - exiting 07:03:40 (2640): No heartbeat from core client for 30 sec - exiting 07:03:41 (2640): No heartbeat from core client for 30 sec - exiting 07:03:42 (2640): No heartbeat from core client for 30 sec - exiting 07:03:43 (2640): No heartbeat from core client for 30 sec - exiting 07:03:44 (2640): No heartbeat from core client for 30 sec - exiting 07:03:45 (2640): No heartbeat from core client for 30 sec - exiting 07:03:46 (2640): No heartbeat from core client for 30 sec - exiting 07:03:48 (2640): No heartbeat from core client for 30 sec - exiting 07:03:49 (2640): No heartbeat from core client for 30 sec - exiting 07:03:50 (2640): No heartbeat from core client for 30 sec - exiting 07:03:51 (2640): No heartbeat from core client for 30 sec - exiting 07:03:52 (2640): No heartbeat from core client for 30 sec - exiting 07:03:53 (2640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77A6E7AA read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... 07:12:21 (3816): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76F15F1B read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Oct 2011 15:38:03	1091933	13405378	hadcm3n_p04i_1940_40_007419374_2	259,200	486,634	1.8774
18 Oct 2011 13:34:12	1091933	13405378	hadcm3n_p04i_1940_40_007419374_2	233,280	437,682	1.8762
17 Oct 2011 06:13:38	1091933	13405378	hadcm3n_p04i_1940_40_007419374_2	207,360	388,609	1.8741
12 Oct 2011 07:39:12	1091933	13405378	hadcm3n_p04i_1940_40_007419374_2	181,440	339,391	1.8705
10 Oct 2011 10:10:48	1091933	13405378	hadcm3n_p04i_1940_40_007419374_2	155,520	291,257	1.8728
05 Oct 2011 12:28:49	1091933	13405378	hadcm3n_p04i_1940_40_007419374_2	129,600	242,711	1.8728
04 Oct 2011 05:38:43	1091933	13405378	hadcm3n_p04i_1940_40_007419374_2	103,680	193,959	1.8707
02 Oct 2011 02:20:39	1091933	13405378	hadcm3n_p04i_1940_40_007419374_2	77,760	145,488	1.8710
01 Oct 2011 12:52:22	1091933	13405378	hadcm3n_p04i_1940_40_007419374_2	51,840	97,572	1.8822
27 Sep 2011 08:58:13	1091933	13405378	hadcm3n_p04i_1940_40_007419374_2	25,920	48,883	1.8859