Task 15752766

Name	hadcm3n_n5q8_1920_40_008323652_2
Workunit	8474787
Created	27 Apr 2013, 2:35:20 UTC
Sent	27 Apr 2013, 2:35:38 UTC
Report deadline	27 Jul 2013, 10:02:49 UTC
Received	29 May 2013, 16:23:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1237050
Run time	24 days 21 hours 31 min 22 sec
CPU time	22 days 19 hours 38 min 59 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 11:45:04 (5692): No heartbeat from core client for 30 sec - exiting 11:45:05 (5692): No heartbeat from core client for 30 sec - exiting 11:45:06 (5692): No heartbeat from core client for 30 sec - exiting 11:45:07 (5692): No heartbeat from core client for 30 sec - exiting 11:45:08 (5692): No heartbeat from core client for 30 sec - exiting 11:45:09 (5692): No heartbeat from core client for 30 sec - exiting 11:45:10 (5692): No heartbeat from core client for 30 sec - exiting 11:45:11 (5692): No heartbeat from core client for 30 sec - exiting 11:45:12 (5692): No heartbeat from core client for 30 sec - exiting 11:45:13 (5692): No heartbeat from core client for 30 sec - exiting 11:45:14 (5692): No heartbeat from core client for 30 sec - exiting 11:45:16 (5692): No heartbeat from core client for 30 sec - exiting 11:45:17 (5692): No heartbeat from core client for 30 sec - exiting 11:45:18 (5692): No heartbeat from core client for 30 sec - exiting 11:45:19 (5692): No heartbeat from core client for 30 sec - exiting 11:45:20 (5692): No heartbeat from core client for 30 sec - exiting 11:45:21 (5692): No heartbeat from core client for 30 sec - exiting 11:45:22 (5692): No heartbeat from core client for 30 sec - exiting 11:45:23 (5692): No heartbeat from core client for 30 sec - exiting 11:45:24 (5692): No heartbeat from core client for 30 sec - exiting 11:45:25 (5692): No heartbeat from core client for 30 sec - exiting 11:45:26 (5692): No heartbeat from core client for 30 sec - exiting 11:45:28 (5692): No heartbeat from core client for 30 sec - exiting 11:45:29 (5692): No heartbeat from core client for 30 sec - exiting 11:45:30 (5692): No heartbeat from core client for 30 sec - exiting 11:45:31 (5692): No heartbeat from core client for 30 sec - exiting 11:45:32 (5692): No heartbeat from core client for 30 sec - exiting 11:45:33 (5692): No heartbeat from core client for 30 sec - exiting 11:45:34 (5692): No heartbeat from core client for 30 sec - exiting 11:45:35 (5692): No heartbeat from core client for 30 sec - exiting 11:45:36 (5692): No heartbeat from core client for 30 sec - exiting 11:45:37 (5692): No heartbeat from core client for 30 sec - exiting 11:45:38 (5692): No heartbeat from core client for 30 sec - exiting 11:45:40 (5692): No heartbeat from core client for 30 sec - exiting 11:45:41 (5692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:45:42 (5692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:57:06 (6376): No heartbeat from core client for 30 sec - exiting 10:57:07 (6376): No heartbeat from core client for 30 sec - exiting 10:57:08 (6376): No heartbeat from core client for 30 sec - exiting 10:57:09 (6376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:57:10 (6376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:09:34 (6984): No heartbeat from core client for 30 sec - exiting 11:09:36 (6984): No heartbeat from core client for 30 sec - exiting 11:09:37 (6984): No heartbeat from core client for 30 sec - exiting 11:09:38 (6984): No heartbeat from core client for 30 sec - exiting 11:09:39 (6984): No heartbeat from core client for 30 sec - exiting 11:09:40 (6984): No heartbeat from core client for 30 sec - exiting 11:09:41 (6984): No heartbeat from core client for 30 sec - exiting 11:09:42 (6984): No heartbeat from core client for 30 sec - exiting 11:09:43 (6984): No heartbeat from core client for 30 sec - exiting 11:09:44 (6984): No heartbeat from core client for 30 sec - exiting 11:09:45 (6984): No heartbeat from core client for 30 sec - exiting 11:09:46 (6984): No heartbeat from core client for 30 sec - exiting 11:09:47 (6984): No heartbeat from core client for 30 sec - exiting 11:09:48 (6984): No heartbeat from core client for 30 sec - exiting 11:09:49 (6984): No heartbeat from core client for 30 sec - exiting 11:09:50 (6984): No heartbeat from core client for 30 sec - exiting 11:09:51 (6984): No heartbeat from core client for 30 sec - exiting 11:09:52 (6984): No heartbeat from core client for 30 sec - exiting 11:09:53 (6984): No heartbeat from core client for 30 sec - exiting 11:09:54 (6984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:15:15 (12120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:05:44 (14204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=64628, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=64628, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:09:50 (1124): No heartbeat from core client for 30 sec - exiting 16:09:51 (1124): No heartbeat from core client for 30 sec - exiting 16:09:52 (1124): No heartbeat from core client for 30 sec - exiting 16:09:53 (1124): No heartbeat from core client for 30 sec - exiting 16:09:54 (1124): No heartbeat from core client for 30 sec - exiting 16:09:55 (1124): No heartbeat from core client for 30 sec - exiting 16:09:56 (1124): No heartbeat from core client for 30 sec - exiting 16:09:57 (1124): No heartbeat from core client for 30 sec - exiting 16:09:58 (1124): No heartbeat from core client for 30 sec - exiting 16:09:59 (1124): No heartbeat from core client for 30 sec - exiting 16:10:00 (1124): No heartbeat from core client for 30 sec - exiting 16:10:01 (1124): No heartbeat from core client for 30 sec - exiting 16:10:02 (1124): No heartbeat from core client for 30 sec - exiting 16:10:03 (1124): No heartbeat from core client for 30 sec - exiting 16:10:04 (1124): No heartbeat from core client for 30 sec - exiting 16:10:05 (1124): No heartbeat from core client for 30 sec - exiting 16:10:06 (1124): No heartbeat from core client for 30 sec - exiting 16:10:07 (1124): No heartbeat from core client for 30 sec - exiting 16:10:08 (1124): No heartbeat from core client for 30 sec - exiting 16:10:09 (1124): No heartbeat from core client for 30 sec - exiting 16:10:10 (1124): No heartbeat from core client for 30 sec - exiting 16:10:11 (1124): No heartbeat from core client for 30 sec - exiting 16:10:12 (1124): No heartbeat from core client for 30 sec - exiting 16:10:13 (1124): No heartbeat from core client for 30 sec - exiting 16:10:14 (1124): No heartbeat from core client for 30 sec - exiting 16:10:15 (1124): No heartbeat from core client for 30 sec - exiting 16:10:16 (1124): No heartbeat from core client for 30 sec - exiting 16:10:17 (1124): No heartbeat from core client for 30 sec - exiting 16:10:18 (1124): No heartbeat from core client for 30 sec - exiting 16:10:19 (1124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:10:20 (1124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9784, iMonCtr=1 Model crash detected, will try to restart... 13:35:29 (5260): No heartbeat from core client for 30 sec - exiting 13:35:31 (5260): No heartbeat from core client for 30 sec - exiting 13:35:32 (5260): No heartbeat from core client for 30 sec - exiting 13:35:33 (5260): No heartbeat from core client for 30 sec - exiting 13:35:34 (5260): No heartbeat from core client for 30 sec - exiting 13:35:35 (5260): No heartbeat from core client for 30 sec - exiting 13:35:36 (5260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:35:38 (5260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 06:43:56 (4048): No heartbeat from core client for 30 sec - exiting 06:43:58 (4048): No heartbeat from core client for 30 sec - exiting 06:43:59 (4048): No heartbeat from core client for 30 sec - exiting 06:44:00 (4048): No heartbeat from core client for 30 sec - exiting 06:44:01 (4048): No heartbeat from core client for 30 sec - exiting 06:44:02 (4048): No heartbeat from core client for 30 sec - exiting 06:44:03 (4048): No heartbeat from core client for 30 sec - exiting 06:44:04 (4048): No heartbeat from core client for 30 sec - exiting 06:44:05 (4048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:17:26 (5812): No heartbeat from core client for 30 sec - exiting 14:17:27 (5812): No heartbeat from core client for 30 sec - exiting 14:17:28 (5812): No heartbeat from core client for 30 sec - exiting 14:17:29 (5812): No heartbeat from core client for 30 sec - exiting 14:17:30 (5812): No heartbeat from core client for 30 sec - exiting 14:17:31 (5812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:17:32 (5812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77003AB3 read attempt to address 0x40B63BFC Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77B33AB3 read attempt to address 0x40B63BFC Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 May 2013 21:17:41	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	1,036,800	1,970,832	1.9009
28 May 2013 05:02:19	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	1,010,880	1,918,082	1.8974
27 May 2013 12:50:23	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	984,960	1,865,655	1.8941
24 May 2013 06:17:03	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	959,040	1,820,114	1.8978
23 May 2013 14:20:00	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	933,120	1,766,321	1.8929
22 May 2013 21:38:28	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	907,200	1,715,695	1.8912
22 May 2013 05:17:03	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	881,280	1,665,968	1.8904
21 May 2013 12:44:28	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	855,360	1,610,459	1.8828
17 May 2013 02:07:43	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	829,440	1,561,319	1.8824
16 May 2013 10:02:45	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	803,520	1,507,746	1.8764
15 May 2013 18:11:05	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	777,600	1,456,994	1.8737
15 May 2013 02:48:28	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	751,680	1,409,687	1.8754
14 May 2013 12:19:59	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	725,760	1,358,271	1.8715
13 May 2013 19:58:40	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	699,840	1,309,279	1.8708
13 May 2013 04:31:12	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	673,920	1,260,237	1.8700
12 May 2013 13:07:29	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	648,000	1,210,458	1.8680
11 May 2013 21:57:01	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	622,080	1,161,066	1.8664
11 May 2013 06:42:22	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	596,160	1,111,175	1.8639
10 May 2013 14:55:26	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	570,240	1,060,207	1.8592
09 May 2013 23:33:28	1237050	15752766	hadcm3n_n5q8_1920_40_008323652_2	544,320	1,011,803	1.8588