Task 16142720

Name	hadcm3n_3l0p_1980_40_008363991_4
Workunit	8514850
Created	12 Dec 2013, 19:54:25 UTC
Sent	12 Dec 2013, 19:55:06 UTC
Report deadline	14 Mar 2014, 3:22:17 UTC
Received	11 Feb 2014, 18:28:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1083083
Run time	13 days 18 hours 9 min 44 sec
CPU time	12 days 22 hours 17 min 5 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.28</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 16:27:25 (5820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6532, iMonCtr=1 Model crash detected, will try to restart... 18:35:45 (5676): No heartbeat from core client for 30 sec - exiting 18:35:46 (5676): No heartbeat from core client for 30 sec - exiting 18:35:47 (5676): No heartbeat from core client for 30 sec - exiting 18:35:48 (5676): No heartbeat from core client for 30 sec - exiting 18:35:49 (5676): No heartbeat from core client for 30 sec - exiting 18:35:50 (5676): No heartbeat from core client for 30 sec - exiting 18:35:51 (5676): No heartbeat from core client for 30 sec - exiting 18:35:52 (5676): No heartbeat from core client for 30 sec - exiting 18:35:53 (5676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:54 (5676): No heartbeat from core client for 30 sec - exiting 18:42:01 (5188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:40:05 (5968): No heartbeat from core client for 30 sec - exiting 13:40:06 (5968): No heartbeat from core client for 30 sec - exiting 13:40:07 (5968): No heartbeat from core client for 30 sec - exiting 13:40:08 (5968): No heartbeat from core client for 30 sec - exiting 13:40:09 (5968): No heartbeat from core client for 30 sec - exiting 13:40:10 (5968): No heartbeat from core client for 30 sec - exiting 13:40:11 (5968): No heartbeat from core client for 30 sec - exiting 13:40:12 (5968): No heartbeat from core client for 30 sec - exiting 13:40:13 (5968): No heartbeat from core client for 30 sec - exiting 13:40:14 (5968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:40:15 (5968): No heartbeat from core client for 30 sec - exiting 16:40:31 (4028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:51:46 (1492): No heartbeat from core client for 30 sec - exiting 18:51:47 (1492): No heartbeat from core client for 30 sec - exiting 18:51:48 (1492): No heartbeat from core client for 30 sec - exiting 18:51:49 (1492): No heartbeat from core client for 30 sec - exiting 18:51:50 (1492): No heartbeat from core client for 30 sec - exiting 18:51:51 (1492): No heartbeat from core client for 30 sec - exiting 18:51:52 (1492): No heartbeat from core client for 30 sec - exiting 18:51:53 (1492): No heartbeat from core client for 30 sec - exiting 18:51:51 (1492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:06:47 (4264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:06:48 (4264): No heartbeat from core client for 30 sec - exiting 19:07:38 (4052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:57:39 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3700, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4124, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 21:53:30 (6612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:18:50 (5944): No heartbeat from core client for 30 sec - exiting 08:18:51 (5944): No heartbeat from core client for 30 sec - exiting 08:18:52 (5944): No heartbeat from core client for 30 sec - exiting 08:18:53 (5944): No heartbeat from core client for 30 sec - exiting 08:18:54 (5944): No heartbeat from core client for 30 sec - exiting 08:18:55 (5944): No heartbeat from core client for 30 sec - exiting 08:18:56 (5944): No heartbeat from core client for 30 sec - exiting 08:18:57 (5944): No heartbeat from core client for 30 sec - exiting 08:18:58 (5944): No heartbeat from core client for 30 sec - exiting 08:18:59 (5944): No heartbeat from core client for 30 sec - exiting 08:19:00 (5944): No heartbeat from core client for 30 sec - exiting 08:19:01 (5944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:41:46 (4596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:10:54 (5732): No heartbeat from core client for 30 sec - exiting 09:10:55 (5732): No heartbeat from core client for 30 sec - exiting 09:10:56 (5732): No heartbeat from core client for 30 sec - exiting 09:10:57 (5732): No heartbeat from core client for 30 sec - exiting 09:10:58 (5732): No heartbeat from core client for 30 sec - exiting 09:10:59 (5732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6036, iMonCtr=1 Model crash detected, will try to restart... 09:38:35 (6036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:17:16 (1772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:24:21 (4692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5504, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:56:19 (1712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5040, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4216, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:16:19 (5960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:16:30 (5448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2320, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2320, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2320, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2320, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3704, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:03:18 (4636): No heartbeat from core client for 30 sec - exiting 08:03:19 (4636): No heartbeat from core client for 30 sec - exiting 08:03:20 (4636): No heartbeat from core client for 30 sec - exiting 08:03:21 (4636): No heartbeat from core client for 30 sec - exiting 08:03:22 (4636): No heartbeat from core client for 30 sec - exiting 08:03:23 (4636): No heartbeat from core client for 30 sec - exiting 08:03:24 (4636): No heartbeat from core client for 30 sec - exiting 08:03:25 (4636): No heartbeat from core client for 30 sec - exiting 08:03:26 (4636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5632, iMonCtr=1 Model crash detected, will try to restart... 17:43:09 (2444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5212, iMonCtr=1 Model crash detected, will try to restart... 11:48:30 (5080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:48:32 (5080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:09:00 (4516): No heartbeat from core client for 30 sec - exiting 21:09:01 (4516): No heartbeat from core client for 30 sec - exiting 21:09:02 (4516): No heartbeat from core client for 30 sec - exiting 21:09:03 (4516): No heartbeat from core client for 30 sec - exiting 21:09:04 (4516): No heartbeat from core client for 30 sec - exiting 21:09:05 (4516): No heartbeat from core client for 30 sec - exiting 21:09:06 (4516): No heartbeat from core client for 30 sec - exiting 21:09:07 (4516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5768, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5768, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4524, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1856, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3360, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3360, iMonCtr=1 Model crash detected, will try to restart... 07:54:10 (5504): No heartbeat from core client for 30 sec - exiting 07:54:11 (5504): No heartbeat from core client for 30 sec - exiting 07:54:12 (5504): No heartbeat from core client for 30 sec - exiting 07:54:13 (5504): No heartbeat from core client for 30 sec - exiting 07:54:14 (5504): No heartbeat from core client for 30 sec - exiting 07:54:15 (5504): No heartbeat from core client for 30 sec - exiting 07:54:16 (5504): No heartbeat from core client for 30 sec - exiting 07:54:17 (5504): No heartbeat from core client for 30 sec - exiting 07:54:18 (5504): No heartbeat from core client for 30 sec - exiting 07:54:19 (5504): No heartbeat from core client for 30 sec - exiting 07:54:20 (5504): No heartbeat from core client for 30 sec - exiting 07:54:21 (5504): No heartbeat from core client for 30 sec - exiting 07:54:22 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:21:42 (5336): No heartbeat from core client for 30 sec - exiting 17:21:43 (5336): No heartbeat from core client for 30 sec - exiting 17:21:44 (5336): No heartbeat from core client for 30 sec - exiting 17:21:45 (5336): No heartbeat from core client for 30 sec - exiting 17:21:46 (5336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76FE3FA9 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5636, selfPID=5636, iMonCtr=1 </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Feb 2014 16:21:48	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	777,600	1,117,004	1.4365
09 Feb 2014 04:34:50	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	751,680	1,079,654	1.4363
08 Feb 2014 17:06:06	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	725,760	1,041,612	1.4352
30 Jan 2014 23:00:34	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	699,840	1,005,304	1.4365
25 Jan 2014 11:42:42	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	673,920	968,326	1.4369
19 Jan 2014 21:57:57	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	648,000	932,052	1.4384
17 Jan 2014 06:56:45	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	622,080	895,804	1.4400
13 Jan 2014 01:56:28	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	596,160	858,796	1.4405
11 Jan 2014 16:33:21	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	570,240	821,466	1.4406
10 Jan 2014 03:00:44	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	544,320	784,700	1.4416
07 Jan 2014 16:19:12	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	518,400	747,961	1.4428
06 Jan 2014 23:25:28	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	492,480	710,682	1.4431
06 Jan 2014 08:43:40	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	466,560	673,326	1.4432
05 Jan 2014 19:39:47	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	440,640	635,305	1.4418
03 Jan 2014 22:26:56	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	414,720	596,718	1.4388
03 Jan 2014 09:50:54	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	388,800	559,431	1.4389
02 Jan 2014 22:23:12	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	362,880	522,081	1.4387
02 Jan 2014 00:18:10	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	336,960	484,174	1.4369
01 Jan 2014 14:00:55	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	311,040	448,183	1.4409
01 Jan 2014 05:03:22	1083083	16142720	hadcm3n_3l0p_1980_40_008363991_4	285,120	413,017	1.4486