Task 16024104

Name	hadcm3n_7yby_1980_40_008455777_3
Workunit	8606633
Created	19 Sep 2013, 7:17:32 UTC
Sent	19 Sep 2013, 7:46:13 UTC
Report deadline	19 Dec 2013, 15:13:24 UTC
Received	20 Oct 2013, 8:33:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1098820
Run time	9 days 19 hours 10 min 16 sec
CPU time	8 days 16 hours 43 min 46 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.51 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.6.20</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 11:15:04 (1444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:35:13 (2644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:56:10 (1020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3180, iMonCtr=1 Model crash detected, will try to restart... 09:09:13 (3128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:21:06 (3568): No heartbeat from core client for 30 sec - exiting 14:21:07 (3568): No heartbeat from core client for 30 sec - exiting 14:21:08 (3568): No heartbeat from core client for 30 sec - exiting 14:21:09 (3568): No heartbeat from core client for 30 sec - exiting 14:21:10 (3568): No heartbeat from core client for 30 sec - exiting 14:21:11 (3568): No heartbeat from core client for 30 sec - exiting 14:21:12 (3568): No heartbeat from core client for 30 sec - exiting 14:21:13 (3568): No heartbeat from core client for 30 sec - exiting 14:21:15 (3568): No heartbeat from core client for 30 sec - exiting 14:21:16 (3568): No heartbeat from core client for 30 sec - exiting 14:21:17 (3568): No heartbeat from core client for 30 sec - exiting 14:21:18 (3568): No heartbeat from core client for 30 sec - exiting 14:21:19 (3568): No heartbeat from core client for 30 sec - exiting 14:21:20 (3568): No heartbeat from core client for 30 sec - exiting 14:21:21 (3568): No heartbeat from core client for 30 sec - exiting 14:21:22 (3568): No heartbeat from core client for 30 sec - exiting 14:21:23 (3568): No heartbeat from core client for 30 sec - exiting 14:21:24 (3568): No heartbeat from core client for 30 sec - exiting 14:21:25 (3568): No heartbeat from core client for 30 sec - exiting 14:21:26 (3568): No heartbeat from core client for 30 sec - exiting 14:21:27 (3568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/7ybyko.pji3c10 Error converting file to netcdf: dataout/7ybyko.pii3c10 Error converting file to netcdf: dataout/7ybyko.pfi3c10 Error converting file to netcdf: dataout/7ybyka.phi3c10 Error converting file to netcdf: dataout/7ybyka.pgi3c10 Error converting file to netcdf: dataout/7ybyka.pei3c10 Error converting file to netcdf: dataout/7ybyka.pdi3c10 18:17:25 (5332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... C10:12:02 (3476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:12:03 (3476): No heartbeat from core client for 30 sec - exiting 10:12:04 (3476): No heartbeat from core client for 30 sec - exiting 10:12:05 (3476): No heartbeat from core client for 30 sec - exiting 10:12:07 (3476): No heartbeat from core client for 30 sec - exiting 10:12:08 (3476): No heartbeat from core client for 30 sec - exiting 10:12:09 (3476): No heartbeat from core client for 30 sec - exiting 10:12:10 (3476): No heartbeat from core client for 30 sec - exiting 14:55:32 (3156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:26:03 (3688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:26:35 (1400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:15:22 (508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:34:32 (1644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:42:46 (2620): No heartbeat from core client for 30 sec - exiting 11:42:47 (2620): No heartbeat from core client for 30 sec - exiting 11:42:48 (2620): No heartbeat from core client for 30 sec - exiting 11:42:49 (2620): No heartbeat from core client for 30 sec - exiting 11:42:50 (2620): No heartbeat from core client for 30 sec - exiting 11:42:51 (2620): No heartbeat from core client for 30 sec - exiting 11:42:52 (2620): No heartbeat from core client for 30 sec - exiting 11:42:53 (2620): No heartbeat from core client for 30 sec - exiting 11:42:54 (2620): No heartbeat from core client for 30 sec - exiting 11:42:56 (2620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:42:57 (2620): No heartbeat from core client for 30 sec - exiting 11:42:58 (2620): No heartbeat from core client for 30 sec - exiting 11:42:59 (2620): No heartbeat from core client for 30 sec - exiting 11:43:00 (2620): No heartbeat from core client for 30 sec - exiting 11:43:01 (2620): No heartbeat from core client for 30 sec - exiting 11:43:04 (2620): No heartbeat from core client for 30 sec - exiting 11:43:05 (2620): No heartbeat from core client for 30 sec - exiting 11:43:06 (2620): No heartbeat from core client for 30 sec - exiting 11:43:07 (2620): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1436, iMonCtr=1 Model crash detected, will try to restart... 09:51:59 (3804): No heartbeat from core client for 30 sec - exiting 09:52:00 (3804): No heartbeat from core client for 30 sec - exiting 09:52:01 (3804): No heartbeat from core client for 30 sec - exiting 09:52:02 (3804): No heartbeat from core client for 30 sec - exiting 09:52:03 (3804): No heartbeat from core client for 30 sec - exiting 09:52:04 (3804): No heartbeat from core client for 30 sec - exiting 09:52:05 (3804): No heartbeat from core client for 30 sec - exiting 09:52:06 (3804): No heartbeat from core client for 30 sec - exiting 09:52:07 (3804): No heartbeat from core client for 30 sec - exiting 09:52:08 (3804): No heartbeat from core client for 30 sec - exiting 09:52:09 (3804): No heartbeat from core client for 30 sec - exiting 09:52:10 (3804): No heartbeat from core client for 30 sec - exiting 09:52:11 (3804): No heartbeat from core client for 30 sec - exiting 09:52:12 (3804): No heartbeat from core client for 30 sec - exiting 09:52:13 (3804): No heartbeat from core client for 30 sec - exiting 09:52:14 (3804): No heartbeat from core client for 30 sec - exiting 09:52:15 (3804): No heartbeat from core client for 30 sec - exiting 09:52:16 (3804): No heartbeat from core client for 30 sec - exiting 09:52:17 (3804): No heartbeat from core client for 30 sec - exiting 09:52:18 (3804): No heartbeat from core client for 30 sec - exiting 09:52:19 (3804): No heartbeat from core client for 30 sec - exiting 09:52:20 (3804): No heartbeat from core client for 30 sec - exiting 09:52:21 (3804): No heartbeat from core client for 30 sec - exiting 09:52:22 (3804): No heartbeat from core client for 30 sec - exiting 09:52:23 (3804): No heartbeat from core client for 30 sec - exiting 09:52:24 (3804): No heartbeat from core client for 30 sec - exiting 09:52:25 (3804): No heartbeat from core client for 30 sec - exiting 09:52:26 (3804): No heartbeat from core client for 30 sec - exiting 09:52:27 (3804): No heartbeat from core client for 30 sec - exiting 09:52:28 (3804): No heartbeat from core client for 30 sec - exiting 09:52:29 (3804): No heartbeat from core client for 30 sec - exiting 09:52:30 (3804): No heartbeat from core client for 30 sec - exiting 09:52:31 (3804): No heartbeat from core client for 30 sec - exiting 09:52:32 (3804): No heartbeat from core client for 30 sec - exiting 09:52:33 (3804): No heartbeat from core client for 30 sec - exiting 09:52:34 (3804): No heartbeat from core client for 30 sec - exiting 09:52:35 (3804): No heartbeat from core client for 30 sec - exiting 09:52:36 (3804): No heartbeat from core client for 30 sec - exiting 09:52:37 (3804): No heartbeat from core client for 30 sec - exiting 09:52:38 (3804): No heartbeat from core client for 30 sec - exiting 09:52:39 (3804): No heartbeat from core client for 30 sec - exiting 09:52:40 (3804): No heartbeat from core client for 30 sec - exiting 09:52:41 (3804): No heartbeat from core client for 30 sec - exiting 09:52:42 (3804): No heartbeat from core client for 30 sec - exiting 09:52:43 (3804): No heartbeat from core client for 30 sec - exiting 09:52:44 (3804): No heartbeat from core client for 30 sec - exiting 09:52:45 (3804): No heartbeat from core client for 30 sec - exiting 09:52:46 (3804): No heartbeat from core client for 30 sec - exiting 09:52:47 (3804): No heartbeat from core client for 30 sec - exiting 09:52:48 (3804): No heartbeat from core client for 30 sec - exiting 09:52:49 (3804): No heartbeat from core client for 30 sec - exiting 09:52:50 (3804): No heartbeat from core client for 30 sec - exiting 09:52:51 (3804): No heartbeat from core client for 30 sec - exiting 09:52:52 (3804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:52:53 (3804): No heartbeat from core client for 30 sec - exiting 09:52:54 (3804): No heartbeat from core client for 30 sec - exiting 09:52:55 (3804): No heartbeat from core client for 30 sec - exiting 09:52:56 (3804): No heartbeat from core client for 30 sec - exiting 09:52:57 (3804): No heartbeat from core client for 30 sec - exiting 09:52:58 (3804): No heartbeat from core client for 30 sec - exiting 09:52:59 (3804): No heartbeat from core client for 30 sec - exiting BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 22:18:12 (3700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:11:52 (2280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:32:33 (2608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2148, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3660, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2288, iMonCtr=1 Model crash detected, will try to restart... 09:48:27 (3464): No heartbeat from core client for 30 sec - exiting 09:48:28 (3464): No heartbeat from core client for 30 sec - exiting 09:48:29 (3464): No heartbeat from core client for 30 sec - exiting 09:48:30 (3464): No heartbeat from core client for 30 sec - exiting 09:48:31 (3464): No heartbeat from core client for 30 sec - exiting 09:48:32 (3464): No heartbeat from core client for 30 sec - exiting 09:48:33 (3464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:48:35 (3464): No heartbeat from core client for 30 sec - exiting 09:48:38 (3464): No heartbeat from core client for 30 sec - exiting 09:48:42 (3464): No heartbeat from core client for 30 sec - exiting 09:48:44 (3464): No heartbeat from core client for 30 sec - exiting 12:11:21 (2292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:11:22 (2292): No heartbeat from core client for 30 sec - exiting 12:11:23 (2292): No heartbeat from core client for 30 sec - exiting 12:11:24 (2292): No heartbeat from core client for 30 sec - exiting 12:11:25 (2292): No heartbeat from core client for 30 sec - exiting 12:11:26 (2292): No heartbeat from core client for 30 sec - exiting 12:11:27 (2292): No heartbeat from core client for 30 sec - exiting 12:11:28 (2292): No heartbeat from core client for 30 sec - exiting 12:11:29 (2292): No heartbeat from core client for 30 sec - exiting 12:11:30 (2292): No heartbeat from core client for 30 sec - exiting 12:11:31 (2292): No heartbeat from core client for 30 sec - exiting 20:52:58 (3856): No heartbeat from core client for 30 sec - exiting 20:52:59 (3856): No heartbeat from core client for 30 sec - exiting 20:53:00 (3856): No heartbeat from core client for 30 sec - exiting 20:53:01 (3856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x773E8755 read attempt to address 0x40E81C8A Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_7yby_1980_40_008455777/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Oct 2013 08:34:22	1098820	16024104	hadcm3n_7yby_1980_40_008455777_3	259,200	751,560	2.8995
15 Oct 2013 17:54:01	1098820	16024104	hadcm3n_7yby_1980_40_008455777_3	233,280	673,507	2.8871
13 Oct 2013 13:23:16	1098820	16024104	hadcm3n_7yby_1980_40_008455777_3	207,360	593,032	2.8599
10 Oct 2013 07:15:12	1098820	16024104	hadcm3n_7yby_1980_40_008455777_3	181,440	516,674	2.8476
07 Oct 2013 08:40:11	1098820	16024104	hadcm3n_7yby_1980_40_008455777_3	155,520	438,661	2.8206
03 Oct 2013 18:53:58	1098820	16024104	hadcm3n_7yby_1980_40_008455777_3	129,600	364,658	2.8137
30 Sep 2013 17:42:59	1098820	16024104	hadcm3n_7yby_1980_40_008455777_3	103,680	301,054	2.9037
28 Sep 2013 12:26:13	1098820	16024104	hadcm3n_7yby_1980_40_008455777_3	77,760	224,824	2.8913
26 Sep 2013 12:28:50	1098820	16024104	hadcm3n_7yby_1980_40_008455777_3	51,840	151,404	2.9206
23 Sep 2013 12:49:27	1098820	16024104	hadcm3n_7yby_1980_40_008455777_3	25,920	76,375	2.9466