Task 16082186

Name	hadcm3n_4e7e_2020_40_008406296_1
Workunit	8557152
Created	19 Nov 2013, 12:44:59 UTC
Sent	19 Nov 2013, 12:45:05 UTC
Report deadline	18 Feb 2014, 20:12:16 UTC
Received	6 Jan 2014, 16:19:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1298358
Run time	13 days 15 hours 6 min 58 sec
CPU time	7 days 14 hours 31 min 46 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.59 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5668, iMonCtr=1 Model crash detected, will try to restart... ConController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4968, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5104, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5340, iMonCtr=1 Model crash detected, will try to restart... 20:08:58 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3228, iMonCtr=1 Model crash detected, will try to restart... 23:13:24 (4144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6068, iMonCtr=1 Model crash detected, will try to restart... 18:26:00 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:15:41 (2424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4484, iMonCtr=1 Model crash detected, will try to restart... 21:44:33 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:50:34 (3796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4796, iMonCtr=1 Model crash detected, will try to restart... 15:09:55 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4988, iMonCtr=1 Model crash detected, will try to restart... 01:21:30 (4248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:38:54 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5524, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... 06:40:24 (3024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5136, iMonCtr=1 Model crash detected, will try to restart... 15:05:40 (4552): No heartbeat from core client for 30 sec - exiting 15:05:41 (4552): No heartbeat from core client for 30 sec - exiting 15:05:42 (4552): No heartbeat from core client for 30 sec - exiting 15:05:43 (4552): No heartbeat from core client for 30 sec - exiting 15:05:44 (4552): No heartbeat from core client for 30 sec - exiting 15:05:46 (4552): No heartbeat from core client for 30 sec - exiting 15:05:47 (4552): No heartbeat from core client for 30 sec - exiting 15:05:48 (4552): No heartbeat from core client for 30 sec - exiting 15:05:49 (4552): No heartbeat from core client for 30 sec - exiting 15:05:50 (4552): No heartbeat from core client for 30 sec - exiting 15:05:51 (4552): No heartbeat from core client for 30 sec - exiting 15:05:52 (4552): No heartbeat from core client for 30 sec - exiting 15:05:53 (4552): No heartbeat from core client for 30 sec - exiting 15:05:54 (4552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4352, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4608, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4272, iMonCtr=1 Model crash detected, will try to restart... 10:08:29 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:39:15 (4408): No heartbeat from core client for 30 sec - exiting 14:39:16 (4408): No heartbeat from core client for 30 sec - exiting 14:39:17 (4408): No heartbeat from core client for 30 sec - exiting 14:39:18 (4408): No heartbeat from core client for 30 sec - exiting 14:39:19 (4408): No heartbeat from core client for 30 sec - exiting 14:39:20 (4408): No heartbeat from core client for 30 sec - exiting 14:39:21 (4408): No heartbeat from core client for 30 sec - exiting 14:39:22 (4408): No heartbeat from core client for 30 sec - exiting 14:39:23 (4408): No heartbeat from core client for 30 sec - exiting 14:39:24 (4408): No heartbeat from core client for 30 sec - exiting 14:39:25 (4408): No heartbeat from core client for 30 sec - exiting 14:39:26 (4408): No heartbeat from core client for 30 sec - exiting 14:39:27 (4408): No heartbeat from core client for 30 sec - exiting 14:39:28 (4408): No heartbeat from core client for 30 sec - exiting 14:39:29 (4408): No heartbeat from core client for 30 sec - exiting 14:39:30 (4408): No heartbeat from core client for 30 sec - exiting 14:39:31 (4408): No heartbeat from core client for 30 sec - exiting 14:39:32 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:39:33 (4408): No heartbeat from core client for 30 sec - exiting 14:39:34 (4408): No heartbeat from core client for 30 sec - exiting 14:39:35 (4408): No heartbeat from core client for 30 sec - exiting 14:39:36 (4408): No heartbeat from core client for 30 sec - exiting 14:39:37 (4408): No heartbeat from core client for 30 sec - exiting 14:39:38 (4408): No heartbeat from core client for 30 sec - exiting 05:25:43 (4400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6092, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3100, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4604, iMonCtr=1 Model crash detected, will try to restart... 11:38:35 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:14:53 (4724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:39:24 (2956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:11:52 (4256): No heartbeat from core client for 30 sec - exiting 14:11:53 (4256): No heartbeat from core client for 30 sec - exiting 14:11:54 (4256): No heartbeat from core client for 30 sec - exiting 14:11:56 (4256): No heartbeat from core client for 30 sec - exiting 14:11:57 (4256): No heartbeat from core client for 30 sec - exiting 14:11:58 (4256): No heartbeat from core client for 30 sec - exiting 14:11:59 (4256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5712, iMonCtr=1 Model crash detected, will try to restart... 07:37:02 (5356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:39:49 (5372): No heartbeat from core client for 30 sec - exiting 08:39:50 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:39:51 (5372): No heartbeat from core client for 30 sec - exiting 18:16:25 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:22:21 (5452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4068, iMonCtr=1 Model crash detected, will try to restart... 10:24:46 (5344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4808, iMonCtr=1 Model crash detected, will try to restart... 03:59:12 (4880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:04:44 (2236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5660, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5160, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1740, iMonCtr=1 Model crash detected, will try to restart... 21:45:11 (1648): No heartbeat from core client for 30 sec - exiting 21:45:12 (1648): No heartbeat from core client for 30 sec - exiting 21:45:13 (1648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6000, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5836, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4952, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4992, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4744, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2720, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4544, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5608, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4696, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7726278D read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_4e7e_2020_40_008406296/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Jan 2014 15:44:43	1298358	16082186	hadcm3n_4e7e_2020_40_008406296_1	259,200	657,089	2.5351
01 Jan 2014 18:56:27	1298358	16082186	hadcm3n_4e7e_2020_40_008406296_1	233,280	590,447	2.5311
24 Dec 2013 08:38:36	1298358	16082186	hadcm3n_4e7e_2020_40_008406296_1	207,360	523,556	2.5249
18 Dec 2013 15:39:38	1298358	16082186	hadcm3n_4e7e_2020_40_008406296_1	181,440	457,792	2.5231
15 Dec 2013 17:44:06	1298358	16082186	hadcm3n_4e7e_2020_40_008406296_1	155,520	391,471	2.5172
10 Dec 2013 23:45:09	1298358	16082186	hadcm3n_4e7e_2020_40_008406296_1	129,600	328,478	2.5346
06 Dec 2013 06:36:53	1298358	16082186	hadcm3n_4e7e_2020_40_008406296_1	103,680	263,886	2.5452
03 Dec 2013 15:04:44	1298358	16082186	hadcm3n_4e7e_2020_40_008406296_1	77,760	200,609	2.5798
29 Nov 2013 00:17:32	1298358	16082186	hadcm3n_4e7e_2020_40_008406296_1	51,840	135,674	2.6172
24 Nov 2013 22:25:41	1298358	16082186	hadcm3n_4e7e_2020_40_008406296_1	25,920	67,178	2.5917