Task 13634374

Name	hadcm3n_o6iq_1900_40_007440574_2
Workunit	7638077
Created	15 Nov 2011, 17:34:45 UTC
Sent	18 Nov 2011, 17:00:30 UTC
Report deadline	18 Feb 2012, 0:27:41 UTC
Received	1 Jan 2012, 13:57:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1124257
Run time	25 days 20 hours 43 min 46 sec
CPU time	24 days 11 hours 37 min 43 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.37 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5432, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 12:56:02 (5316): Can't acquire lockfile (32) - waiting 35s No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6852, selfPID=6852, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o6iqko.pja4c10 Error converting file to netcdf: dataout/o6iqko.pia4c10 Error converting file to netcdf: dataout/o6iqko.pfa4c10 Error converting file to netcdf: dataout/o6iqka.pha4c10 Error converting file to netcdf: dataout/o6iqka.pga4c10 Error converting file to netcdf: dataout/o6iqka.pea4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:37:18 (784): No heartbeat from core client for 30 sec - exiting 08:37:19 (784): No heartbeat from core client for 30 sec - exiting 08:37:20 (784): No heartbeat from core client for 30 sec - exiting 08:37:22 (784): No heartbeat from core client for 30 sec - exiting 08:37:23 (784): No heartbeat from core client for 30 sec - exiting 08:37:24 (784): No heartbeat from core client for 30 sec - exiting 08:37:25 (784): No heartbeat from core client for 30 sec - exiting 08:37:26 (784): No heartbeat from core client for 30 sec - exiting 08:37:27 (784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3204, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:16:04 (3316): No heartbeat from core client for 30 sec - exiting 23:16:05 (3316): No heartbeat from core client for 30 sec - exiting 23:16:06 (3316): No heartbeat from core client for 30 sec - exiting 23:16:07 (3316): No heartbeat from core client for 30 sec - exiting 23:16:08 (3316): No heartbeat from core client for 30 sec - exiting 23:16:09 (3316): No heartbeat from core client for 30 sec - exiting 23:16:10 (3316): No heartbeat from core client for 30 sec - exiting 23:16:11 (3316): No heartbeat from core client for 30 sec - exiting 23:16:12 (3316): No heartbeat from core client for 30 sec - exiting 23:16:13 (3316): No heartbeat from core client for 30 sec - exiting 23:16:15 (3316): No heartbeat from core client for 30 sec - exiting 23:16:16 (3316): No heartbeat from core client for 30 sec - exiting 23:16:17 (3316): No heartbeat from core client for 30 sec - exiting 23:16:18 (3316): No heartbeat from core client for 30 sec - exiting 23:16:19 (3316): No heartbeat from core client for 30 sec - exiting 23:16:20 (3316): No heartbeat from core client for 30 sec - exiting 23:16:21 (3316): No heartbeat from core client for 30 sec - exiting 23:16:22 (3316): No heartbeat from core client for 30 sec - exiting 23:16:23 (3316): No heartbeat from core client for 30 sec - exiting 23:16:24 (3316): No heartbeat from core client for 30 sec - exiting 23:16:25 (3316): No heartbeat from core client for 30 sec - exiting 23:16:27 (3316): No heartbeat from core client for 30 sec - exiting 23:16:28 (3316): No heartbeat from core client for 30 sec - exiting 23:16:29 (3316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:16:30 (3316): No heartbeat from core client for 30 sec - exiting 23:16:31 (3316): No heartbeat from core client for 30 sec - exiting 23:16:32 (3316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:01:15 (4988): No heartbeat from core client for 30 sec - exiting 09:01:16 (4988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:35:50 (4344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x778771F5 read attempt to address 0x403808FC Engaging BOINC Windows Runtime Debugger... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=15440, selfPID=15440, iMonCtr=1 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77873A93 read attempt to address 0x403808FC Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Dec 2011 22:09:52	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	777,600	2,088,437	2.6857
31 Dec 2011 06:40:57	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	751,680	2,048,893	2.7258
30 Dec 2011 19:36:02	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	725,760	2,009,483	2.7688
30 Dec 2011 07:49:44	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	699,840	1,967,357	2.8112
29 Dec 2011 15:56:05	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	673,920	1,911,499	2.8364
28 Dec 2011 22:40:45	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	648,000	1,851,133	2.8567
27 Dec 2011 20:26:35	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	622,080	1,777,203	2.8569
25 Dec 2011 14:15:01	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	596,160	1,680,476	2.8188
23 Dec 2011 19:40:58	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	570,240	1,619,225	2.8396
23 Dec 2011 03:04:45	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	544,320	1,562,180	2.8700
22 Dec 2011 09:28:50	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	518,400	1,508,007	2.9090
20 Dec 2011 17:51:11	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	492,480	1,440,938	2.9259
19 Dec 2011 10:07:58	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	466,560	1,371,542	2.9397
18 Dec 2011 00:30:53	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	440,640	1,300,870	2.9522
16 Dec 2011 20:34:56	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	414,720	1,227,350	2.9595
15 Dec 2011 18:40:35	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	388,800	1,156,013	2.9733
14 Dec 2011 15:55:06	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	362,880	1,073,977	2.9596
13 Dec 2011 13:57:51	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	336,960	980,863	2.9109
11 Dec 2011 11:32:27	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	311,040	881,461	2.8339
07 Dec 2011 00:12:49	1124257	13634374	hadcm3n_o6iq_1900_40_007440574_2	285,120	766,887	2.6897