Task 15903237

Name	hadcm3n_o1hi_2020_40_008403285_1
Workunit	8554141
Created	23 Jul 2013, 13:04:41 UTC
Sent	23 Jul 2013, 13:28:45 UTC
Report deadline	22 Oct 2013, 20:55:56 UTC
Received	4 Oct 2013, 16:19:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1286653
Run time	20 days 6 hours 22 min 5 sec
CPU time	17 days 4 hours 32 min 39 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	1.72 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:37:53 (3936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o1hiko.pjm1c10 Error converting file to netcdf: dataout/o1hiko.pim1c10 Error converting file to netcdf: dataout/o1hiko.pfm1c10 Error converting file to netcdf: dataout/o1hika.phm1c10 Error converting file to netcdf: dataout/o1hika.pgm1c10 Error converting file to netcdf: dataout/o1hika.pem1c10 Error converting file to netcdf: dataout/o1hika.pdm1c10 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8008, iMonCtr=1 Model crash detCPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6496, iMonCtr=1 Model crash detected, will try to restart... 15:54:21 (4384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4540, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13180, iMonCtr=1 Model crash detected, will try to restart... 13:38:17 (7152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:46:56 (11816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:46:57 (11816): No heartbeat from core client for 30 sec - exiting 20:46:58 (11816): No heartbeat from core client for 30 sec - exiting 20:46:59 (11816): No heartbeat from core client for 30 sec - exiting 20:47:00 (11816): No heartbeat from core client for 30 sec - exiting 20:47:01 (11816): No heartbeat from core client for 30 sec - exiting 20:47:02 (11816): No heartbeat from core client for 30 sec - exiting 20:47:03 (11816): No heartbeat from core client for 30 sec - exiting 20:47:04 (11816): No heartbeat from core client for 30 sec - exiting 20:47:05 (11816): No heartbeat from core client for 30 sec - exiting 20:47:06 (11816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:06:55 (6528): No heartbeat from core client for 30 sec - exiting 07:06:57 (6528): No heartbeat from core client for 30 sec - exiting 07:06:58 (6528): No heartbeat from core client for 30 sec - exiting 07:06:59 (6528): No heartbeat from core client for 30 sec - exiting 07:07:00 (6528): No heartbeat from core client for 30 sec - exiting 07:07:01 (6528): No heartbeat from core client for 30 sec - exiting 07:07:02 (6528): No heartbeat from core client for 30 sec - exiting 07:07:03 (6528): No heartbeat from core client for 30 sec - exiting 07:07:04 (6528): No heartbeat from core client for 30 sec - exiting 07:07:05 (6528): No heartbeat from core client for 30 sec - exiting 07:07:06 (6528): No heartbeat from core client for 30 sec - exiting 07:07:07 (6528): No heartbeat from core client for 30 sec - exiting 07:07:09 (6528): No heartbeat from core client for 30 sec - exiting 07:07:10 (6528): No heartbeat from core client for 30 sec - exiting 07:07:11 (6528): No heartbeat from core client for 30 sec - exiting 07:07:12 (6528): No heartbeat from core client for 30 sec - exiting 07:07:13 (6528): No heartbeat from core client for 30 sec - exiting 07:07:14 (6528): No heartbeat from core client for 30 sec - exiting 07:07:15 (6528): No heartbeat from core client for 30 sec - exiting 07:07:16 (6528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:53:31 (7500): No heartbeat from core client for 30 sec - exiting 06:53:32 (7500): No heartbeat from core client for 30 sec - exiting 06:53:33 (7500): No heartbeat from core client for 30 sec - exiting 06:53:35 (7500): No heartbeat from core client for 30 sec - exiting 06:53:36 (7500): No heartbeat from core client for 30 sec - exiting 06:53:37 (7500): No heartbeat from core client for 30 sec - exiting 06:53:38 (7500): No heartbeat from core client for 30 sec - exiting 06:53:39 (7500): No heartbeat from core client for 30 sec - exiting 06:53:40 (7500): No heartbeat from core client for 30 sec - exiting 06:53:41 (7500): No heartbeat from core client for 30 sec - exiting 06:53:42 (7500): No heartbeat from core client for 30 sec - exiting 06:53:43 (7500): No heartbeat from core client for 30 sec - exiting 06:53:44 (7500): No heartbeat from core client for 30 sec - exiting 06:53:45 (7500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1552, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7436, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... C12:37:29 (6116): No heartbeat from core client for 30 sec - exiting 12:37:31 (6116): No heartbeat from core client for 30 sec - exiting 12:37:32 (6116): No heartbeat from core client for 30 sec - exiting 12:37:33 (6116): No heartbeat from core client for 30 sec - exiting 12:37:34 (6116): No heartbeat from core client for 30 sec - exiting 12:37:35 (6116): No heartbeat from core client for 30 sec - exiting 12:37:36 (6116): No heartbeat from core client for 30 sec - exiting 12:37:37 (6116): No heartbeat from core client for 30 sec - exiting 12:37:38 (6116): No heartbeat from core client for 30 sec - exiting 12:37:39 (6116): No heartbeat from core client for 30 sec - exiting 12:37:41 (6116): No heartbeat from core client for 30 sec - exiting 12:37:42 (6116): No heartbeat from core client for 30 sec - exiting 12:37:43 (6116): No heartbeat from core client for 30 sec - exiting 12:37:44 (6116): No heartbeat from core client for 30 sec - exiting 12:37:45 (6116): No heartbeat from core client for 30 sec - exiting 12:37:46 (6116): No heartbeat from core client for 30 sec - exiting 12:37:47 (6116): No heartbeat from core client for 30 sec - exiting 12:37:48 (6116): No heartbeat from core client for 30 sec - exiting 12:37:49 (6116): No heartbeat from core client for 30 sec - exiting 12:37:50 (6116): No heartbeat from core client for 30 sec - exiting 12:37:51 (6116): No heartbeat from core client for 30 sec - exiting 12:37:53 (6116): No heartbeat from core client for 30 sec - exiting 12:37:54 (6116): No heartbeat from core client for 30 sec - exiting 12:37:55 (6116): No heartbeat from core client for 30 sec - exiting 12:37:56 (6116): No heartbeat from core client for 30 sec - exiting 12:37:57 (6116): No heartbeat from core client for 30 sec - exiting 12:37:58 (6116): No heartbeat from core client for 30 sec - exiting 12:37:59 (6116): No heartbeat from core client for 30 sec - exiting 12:38:00 (6116): No heartbeat from core client for 30 sec - exiting 12:38:01 (6116): No heartbeat from core client for 30 sec - exiting 12:38:03 (6116): No heartbeat from core client for 30 sec - exiting 12:38:04 (6116): No heartbeat from core client for 30 sec - exiting 12:38:05 (6116): No heartbeat from core client for 30 sec - exiting 12:38:06 (6116): No heartbeat from core client for 30 sec - exiting 12:38:07 (6116): No heartbeat from core client for 30 sec - exiting 12:38:08 (6116): No heartbeat from core client for 30 sec - exiting 12:38:09 (6116): No heartbeat from core client for 30 sec - exiting 12:38:10 (6116): No heartbeat from core client for 30 sec - exiting 12:38:11 (6116): No heartbeat from core client for 30 sec - exiting 12:38:12 (6116): No heartbeat from core client for 30 sec - exiting 12:38:13 (6116): No heartbeat from core client for 30 sec - exiting 12:38:15 (6116): No heartbeat from core client for 30 sec - exiting 12:38:16 (6116): No heartbeat from core client for 30 sec - exiting 12:38:17 (6116): No heartbeat from core client for 30 sec - exiting 12:38:18 (6116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:25 (7372): No heartbeat from core client for 30 sec - exiting 07:09:26 (7372): No heartbeat from core client for 30 sec - exiting 07:09:27 (7372): No heartbeat from core client for 30 sec - exiting 07:09:29 (7372): No heartbeat from core client for 30 sec - exiting 07:09:30 (7372): No heartbeat from core client for 30 sec - exiting 07:09:31 (7372): No heartbeat from core client for 30 sec - exiting 07:09:32 (7372): No heartbeat from core client for 30 sec - exiting 07:09:33 (7372): No heartbeat from core client for 30 sec - exiting 07:09:34 (7372): No heartbeat from core client for 30 sec - exiting 07:09:35 (7372): No heartbeat from core client for 30 sec - exiting 07:09:36 (7372): No heartbeat from core client for 30 sec - exiting 07:09:37 (7372): No heartbeat from core client for 30 sec - exiting 07:09:38 (7372): No heartbeat from core client for 30 sec - exiting 07:09:39 (7372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:41 (7372): No heartbeat from core client for 30 sec - exiting 07:13:08 (6520): No heartbeat from core client for 30 sec - exiting 07:13:09 (6520): No heartbeat from core client for 30 sec - exiting 07:13:10 (6520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:13:11 (6520): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77D83AC3 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o1hi_2020_40_008403285/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Oct 2013 16:13:56	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	1,036,800	1,485,152	1.4324
03 Oct 2013 20:04:42	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	1,010,880	1,449,153	1.4336
03 Oct 2013 00:52:33	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	984,960	1,412,601	1.4342
02 Oct 2013 04:37:06	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	959,040	1,376,129	1.4349
01 Oct 2013 16:35:11	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	933,120	1,339,671	1.4357
30 Sep 2013 20:14:50	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	907,200	1,299,498	1.4324
30 Sep 2013 08:35:13	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	881,280	1,261,599	1.4316
26 Sep 2013 05:25:49	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	855,360	1,226,041	1.4334
25 Sep 2013 09:25:38	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	829,440	1,187,416	1.4316
24 Sep 2013 01:50:32	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	803,520	1,150,277	1.4315
23 Sep 2013 14:10:26	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	777,600	1,113,147	1.4315
23 Sep 2013 13:09:57	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	751,680	1,074,840	1.4299
21 Sep 2013 10:40:59	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	725,760	1,034,648	1.4256
20 Sep 2013 19:06:14	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	699,840	992,030	1.4175
20 Sep 2013 00:06:00	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	673,920	955,935	1.4185
19 Sep 2013 03:00:56	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	648,000	919,177	1.4185
18 Sep 2013 14:11:16	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	622,080	880,274	1.4150
17 Sep 2013 17:42:57	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	596,160	841,782	1.4120
17 Sep 2013 06:17:20	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	570,240	803,624	1.4093
10 Sep 2013 18:27:02	1286653	15903237	hadcm3n_o1hi_2020_40_008403285_1	544,320	765,001	1.4054