Task 16036197

Name	hadcm3n_o81j_1900_40_008465546_0
Workunit	8616385
Created	27 Sep 2013, 9:16:24 UTC
Sent	9 Oct 2013, 23:38:41 UTC
Report deadline	9 Jan 2014, 7:05:52 UTC
Received	25 Oct 2013, 20:57:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1288758
Run time	5 days 3 hours 59 min 23 sec
CPU time	5 days 2 hours 52 min 42 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.18 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:17:21 (5460): No heartbeat from core client for 30 sec - exiting 08:17:23 (5460): No heartbeat from core client for 30 sec - exiting 08:17:24 (5460): No heartbeat from core client for 30 sec - exiting 08:17:25 (5460): No heartbeat from core client for 30 sec - exiting 08:17:26 (5460): No heartbeat from core client for 30 sec - exiting 08:17:27 (5460): No heartbeat from core client for 30 sec - exiting 08:17:28 (5460): No heartbeat from core client for 30 sec - exiting 08:17:29 (5460): No heartbeat from core client for 30 sec - exiting 08:17:31 (5460): No heartbeat from core client for 30 sec - exiting 08:17:32 (5460): No heartbeat from core client for 30 sec - exiting 08:17:33 (5460): No heartbeat from core client for 30 sec - exiting 08:17:34 (5460): No heartbeat from core client for 30 sec - exiting 08:17:35 (5460): No heartbeat from core client for 30 sec - exiting 08:17:36 (5460): No heartbeat from core client for 30 sec - exiting 08:17:37 (5460): No heartbeat from core client for 30 sec - exiting 08:17:38 (5460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:17:39 (5460): No heartbeat from core client for 30 sec - exiting 08:17:40 (5460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:10:51 (6040): No heartbeat from core client for 30 sec - exiting 12:10:52 (6040): No heartbeat from core client for 30 sec - exiting 12:10:53 (6040): No heartbeat from core client for 30 sec - exiting 12:10:54 (6040): No heartbeat from core client for 30 sec - exiting 12:10:55 (6040): No heartbeat from core client for 30 sec - exiting 12:10:56 (6040): No heartbeat from core client for 30 sec - exiting 12:10:57 (6040): No heartbeat from core client for 30 sec - exiting 12:10:58 (6040): No heartbeat from core client for 30 sec - exiting 12:10:59 (6040): No heartbeat from core client for 30 sec - exiting 12:11:01 (6040): No heartbeat from core client for 30 sec - exiting 12:11:02 (6040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:37:53 (5532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:33:52 (4100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:05:14 (5592): No heartbeat from core client for 30 sec - exiting 08:05:16 (5592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:28:20 (2208): No heartbeat from core client for 30 sec - exiting 08:28:21 (2208): No heartbeat from core client for 30 sec - exiting 08:28:23 (2208): No heartbeat from core client for 30 sec - exiting 08:28:24 (2208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:25 (2208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 08:53:32 (4504): No heartbeat from core client for 30 sec - exiting 08:53:33 (4504): No heartbeat from core client for 30 sec - exiting 08:53:34 (4504): No heartbeat from core client for 30 sec - exiting 08:53:35 (4504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:47:50 (4708): No heartbeat from core client for 30 sec - exiting 09:47:51 (4708): No heartbeat from core client for 30 sec - exiting 09:47:52 (4708): No heartbeat from core client for 30 sec - exiting 09:47:53 (4708): No heartbeat from core client for 30 sec - exiting 09:47:54 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:47:55 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:06:02 (5784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:47:18 (932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o81jko.pja7c10 Error converting file to netcdf: dataout/o81jko.pia7c10 Error converting file to netcdf: dataout/o81jko.pfa7c10 Error converting file to netcdf: dataout/o81jka.pha7c10 Error converting file to netcdf: dataout/o81jka.pga7c10 Error converting file to netcdf: dataout/o81jka.pea7c10 Error converting file to netcdf: dataout/o81jka.pda7c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:18:04 (2928): No heartbeat from core client for 30 sec - exiting 09:18:05 (2928): No heartbeat from core client for 30 sec - exiting 09:18:06 (2928): No heartbeat from core client for 30 sec - exiting 09:18:07 (2928): No heartbeat from core client for 30 sec - exiting 09:18:08 (2928): No heartbeat from core client for 30 sec - exiting 09:18:09 (2928): No heartbeat from core client for 30 sec - exiting 09:18:10 (2928): No heartbeat from core client for 30 sec - exiting 09:18:11 (2928): No heartbeat from core client for 30 sec - exiting 09:18:13 (2928): No heartbeat from core client for 30 sec - exiting 09:18:14 (2928): No heartbeat from core client for 30 sec - exiting 09:18:15 (2928): No heartbeat from core client for 30 sec - exiting 09:18:16 (2928): No heartbeat from core client for 30 sec - exiting 09:18:17 (2928): No heartbeat from core client for 30 sec - exiting 09:18:18 (2928): No heartbeat from core client for 30 sec - exiting 09:18:19 (2928): No heartbeat from core client for 30 sec - exiting 09:18:20 (2928): No heartbeat from core client for 30 sec - exiting 09:18:21 (2928): No heartbeat from core client for 30 sec - exiting 09:18:22 (2928): No heartbeat from core client for 30 sec - exiting 09:18:23 (2928): No heartbeat from core client for 30 sec - exiting 09:18:25 (2928): No heartbeat from core client for 30 sec - exiting 09:18:26 (2928): No heartbeat from core client for 30 sec - exiting 09:18:27 (2928): No heartbeat from core client for 30 sec - exiting 09:18:28 (2928): No heartbeat from core client for 30 sec - exiting 09:18:29 (2928): No heartbeat from core client for 30 sec - exiting 09:18:30 (2928): No heartbeat from core client for 30 sec - exiting 09:18:31 (2928): No heartbeat from core client for 30 sec - exiting 09:18:32 (2928): No heartbeat from core client for 30 sec - exiting 09:18:33 (2928): No heartbeat from core client for 30 sec - exiting 09:18:34 (2928): No heartbeat from core client for 30 sec - exiting 09:18:35 (2928): No heartbeat from core client for 30 sec - exiting 09:18:37 (2928): No heartbeat from core client for 30 sec - exiting 09:18:38 (2928): No heartbeat from core client for 30 sec - exiting 09:18:39 (2928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:29:52 (5116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:25:03 (3784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:37 (5304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:25:06 (3948): No heartbeat from core client for 30 sec - exiting 08:25:07 (3948): No heartbeat from core client for 30 sec - exiting 08:25:08 (3948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6780, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77B23AC3 read attempt to address 0x40BB26D8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77B23114 read attempt to address 0x430119E3 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Oct 2013 14:10:43	1288758	16036197	hadcm3n_o81j_1900_40_008465546_0	259,200	417,875	1.6122
24 Oct 2013 02:21:00	1288758	16036197	hadcm3n_o81j_1900_40_008465546_0	233,280	377,395	1.6178
22 Oct 2013 01:55:21	1288758	16036197	hadcm3n_o81j_1900_40_008465546_0	207,360	336,789	1.6242
19 Oct 2013 18:23:08	1288758	16036197	hadcm3n_o81j_1900_40_008465546_0	181,440	296,463	1.6339
19 Oct 2013 06:39:48	1288758	16036197	hadcm3n_o81j_1900_40_008465546_0	155,520	256,042	1.6464
18 Oct 2013 16:33:14	1288758	16036197	hadcm3n_o81j_1900_40_008465546_0	129,600	214,314	1.6537
17 Oct 2013 17:23:21	1288758	16036197	hadcm3n_o81j_1900_40_008465546_0	103,680	171,694	1.6560
16 Oct 2013 17:25:46	1288758	16036197	hadcm3n_o81j_1900_40_008465546_0	77,760	128,677	1.6548
13 Oct 2013 15:40:51	1288758	16036197	hadcm3n_o81j_1900_40_008465546_0	51,840	86,114	1.6611
11 Oct 2013 14:06:47	1288758	16036197	hadcm3n_o81j_1900_40_008465546_0	25,920	43,290	1.6701