Task 15917582

Name	hadcm3n_o2r6_1940_40_008404541_1
Workunit	8555397
Created	14 Aug 2013, 15:38:26 UTC
Sent	15 Aug 2013, 1:50:12 UTC
Report deadline	14 Nov 2013, 9:17:23 UTC
Received	25 Aug 2013, 22:10:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1227838
Run time	8 days 15 hours 56 min 24 sec
CPU time	8 days 15 hours 12 min 19 sec
Validate state	Invalid
Credit	8,087.04
Device peak FLOPS	2.78 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6404, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1704, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1704, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1108, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1108, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:54:27 (6072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:00:02 (1364): No heartbeat from core client for 30 sec - exiting 08:00:03 (1364): No heartbeat from core client for 30 sec - exiting 08:00:04 (1364): No heartbeat from core client for 30 sec - exiting 08:00:05 (1364): No heartbeat from core client for 30 sec - exiting 08:00:06 (1364): No heartbeat from core client for 30 sec - exiting 08:00:07 (1364): No heartbeat from core client for 30 sec - exiting 08:00:08 (1364): No heartbeat from core client for 30 sec - exiting 08:00:09 (1364): No heartbeat from core client for 30 sec - exiting 08:00:10 (1364): No heartbeat from core client for 30 sec - exiting 08:00:11 (1364): No heartbeat from core client for 30 sec - exiting 08:00:12 (1364): No heartbeat from core client for 30 sec - exiting 08:00:14 (1364): No heartbeat from core client for 30 sec - exiting 08:00:15 (1364): No heartbeat from core client for 30 sec - exiting 08:00:16 (1364): No heartbeat from core client for 30 sec - exiting 08:00:17 (1364): No heartbeat from core client for 30 sec - exiting 08:00:18 (1364): No heartbeat from core client for 30 sec - exiting 08:00:19 (1364): No heartbeat from core client for 30 sec - exiting 08:00:20 (1364): No heartbeat from core client for 30 sec - exiting 08:00:21 (1364): No heartbeat from core client for 30 sec - exiting 08:00:22 (1364): No heartbeat from core client for 30 sec - exiting 08:00:23 (1364): No heartbeat from core client for 30 sec - exiting 08:00:24 (1364): No heartbeat from core client for 30 sec - exiting 08:00:26 (1364): No heartbeat from core client for 30 sec - exiting 08:00:27 (1364): No heartbeat from core client for 30 sec - exiting 08:00:28 (1364): No heartbeat from core client for 30 sec - exiting 08:00:29 (1364): No heartbeat from core client for 30 sec - exiting 08:00:30 (1364): No heartbeat from core client for 30 sec - exiting 08:00:31 (1364): No heartbeat from core client for 30 sec - exiting 08:00:32 (1364): No heartbeat from core client for 30 sec - exiting 08:00:33 (1364): No heartbeat from core client for 30 sec - exiting 08:00:34 (1364): No heartbeat from core client for 30 sec - exiting 08:00:35 (1364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:56:29 (3944): No heartbeat from core client for 30 sec - exiting 11:56:31 (3944): No heartbeat from core client for 30 sec - exiting 11:56:32 (3944): No heartbeat from core client for 30 sec - exiting 11:56:33 (3944): No heartbeat from core client for 30 sec - exiting 11:56:34 (3944): No heartbeat from core client for 30 sec - exiting 11:56:35 (3944): No heartbeat from core client for 30 sec - exiting 11:56:36 (3944): No heartbeat from core client for 30 sec - exiting 11:56:37 (3944): No heartbeat from core client for 30 sec - exiting 11:56:38 (3944): No heartbeat from core client for 30 sec - exiting 11:56:39 (3944): No heartbeat from core client for 30 sec - exiting 11:56:40 (3944): No heartbeat from core client for 30 sec - exiting 11:56:41 (3944): No heartbeat from core client for 30 sec - exiting 11:56:43 (3944): No heartbeat from core client for 30 sec - exiting 11:56:44 (3944): No heartbeat from core client for 30 sec - exiting 11:56:45 (3944): No heartbeat from core client for 30 sec - exiting 11:56:46 (3944): No heartbeat from core client for 30 sec - exiting 11:56:47 (3944): No heartbeat from core client for 30 sec - exiting 11:56:48 (3944): No heartbeat from core client for 30 sec - exiting 11:56:49 (3944): No heartbeat from core client for 30 sec - exiting 11:56:50 (3944): No heartbeat from core client for 30 sec - exiting 11:56:51 (3944): No heartbeat from core client for 30 sec - exiting 11:56:52 (3944): No heartbeat from core client for 30 sec - exiting 11:56:53 (3944): No heartbeat from core client for 30 sec - exiting 11:56:55 (3944): No heartbeat from core client for 30 sec - exiting 11:56:56 (3944): No heartbeat from core client for 30 sec - exiting 11:56:57 (3944): No heartbeat from core client for 30 sec - exiting 11:56:58 (3944): No heartbeat from core client for 30 sec - exiting 11:56:59 (3944): No heartbeat from core client for 30 sec - exiting 11:57:00 (3944): No heartbeat from core client for 30 sec - exiting 11:57:01 (3944): No heartbeat from core client for 30 sec - exiting 11:57:02 (3944): No heartbeat from core client for 30 sec - exiting 11:57:03 (3944): No heartbeat from core client for 30 sec - exiting 11:57:04 (3944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x771E3AC3 read attempt to address 0x4087633D Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Aug 2013 20:30:31	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	673,920	745,939	1.1069
23 Aug 2013 16:38:15	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	518,400	560,690	1.0816
23 Aug 2013 10:11:35	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	492,480	531,604	1.0794
23 Aug 2013 10:11:35	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	466,560	502,675	1.0774
23 Aug 2013 10:11:35	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	440,640	473,864	1.0754
22 Aug 2013 07:59:21	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	414,720	444,735	1.0724
21 Aug 2013 08:01:36	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	388,800	416,079	1.0702
21 Aug 2013 00:03:25	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	362,880	387,961	1.0691
20 Aug 2013 16:15:21	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	336,960	360,004	1.0684
20 Aug 2013 08:13:01	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	311,040	331,568	1.0660
20 Aug 2013 00:04:32	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	285,120	302,980	1.0626
19 Aug 2013 16:16:15	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	259,200	274,781	1.0601
19 Aug 2013 07:29:27	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	233,280	246,318	1.0559
18 Aug 2013 23:52:43	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	207,360	218,975	1.0560
18 Aug 2013 16:16:18	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	181,440	191,827	1.0572
18 Aug 2013 08:29:42	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	155,520	163,890	1.0538
18 Aug 2013 00:37:49	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	129,600	135,831	1.0481
17 Aug 2013 16:56:18	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	103,680	108,177	1.0434
17 Aug 2013 08:44:34	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	77,760	79,506	1.0225
15 Aug 2013 16:33:36	1227838	15917582	hadcm3n_o2r6_1940_40_008404541_1	51,840	52,634	1.0153