Task 15924208

Name	hadcm3n_n26h_1920_40_008371713_1
Workunit	8522572
Created	17 Aug 2013, 2:22:08 UTC
Sent	17 Aug 2013, 2:22:18 UTC
Report deadline	16 Nov 2013, 9:49:29 UTC
Received	11 Sep 2013, 5:46:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1137800
Run time	3 days 2 hours 43 min 13 sec
CPU time	2 days 19 hours 49 min 55 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	2.26 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.5</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 19:33:42 (481824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:44 (480744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:46 (480744): No heartbeat from core client for 30 sec - exiting 20:03:47 (480744): No heartbeat from core client for 30 sec - exiting 20:03:48 (480744): No heartbeat from core client for 30 sec - exiting 20:03:49 (480744): No heartbeat from core client for 30 sec - exiting 20:08:45 (481336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:08:46 (481336): No heartbeat from core client for 30 sec - exiting 20:13:44 (481780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:13:46 (481780): No heartbeat from core client for 30 sec - exiting 20:13:47 (481780): No heartbeat from core client for 30 sec - exiting 20:43:49 (482424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:43:51 (482424): No heartbeat from core client for 30 sec - exiting 20:43:53 (482424): No heartbeat from core client for 30 sec - exiting 20:48:47 (484172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:03:50 (483636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:03:53 (483636): No heartbeat from core client for 30 sec - exiting 21:03:54 (483636): No heartbeat from core client for 30 sec - exiting 21:03:55 (483636): No heartbeat from core client for 30 sec - exiting 21:03:56 (483636): No heartbeat from core client for 30 sec - exiting 21:03:57 (483636): No heartbeat from core client for 30 sec - exiting 21:08:49 (484076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:58:53 (484600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:58:56 (484600): No heartbeat from core client for 30 sec - exiting 22:43:58 (481920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:44:00 (481920): No heartbeat from core client for 30 sec - exiting 22:44:01 (481920): No heartbeat from core client for 30 sec - exiting 22:49:00 (488012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:49:02 (488012): No heartbeat from core client for 30 sec - exiting 22:49:03 (488012): No heartbeat from core client for 30 sec - exiting 22:49:04 (488012): No heartbeat from core client for 30 sec - exiting 22:49:05 (488012): No heartbeat from core client for 30 sec - exiting 23:04:43 (488596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:04:45 (488596): No heartbeat from core client for 30 sec - exiting 23:04:46 (488596): No heartbeat from core client for 30 sec - exiting 23:04:47 (488596): No heartbeat from core client for 30 sec - exiting 23:09:01 (488520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:09:02 (488520): No heartbeat from core client for 30 sec - exiting 23:09:03 (488520): No heartbeat from core client for 30 sec - exiting 23:14:05 (488016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:03 (489244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:04 (489244): No heartbeat from core client for 30 sec - exiting 23:19:05 (489244): No heartbeat from core client for 30 sec - exiting 23:24:05 (489944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 23:29:07 (488316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:29:08 (488316): No heartbeat from core client for 30 sec - exiting 23:29:09 (488316): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 23:54:07 (490060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:54:09 (490060): No heartbeat from core client for 30 sec - exiting 23:54:10 (490060): No heartbeat from core client for 30 sec - exiting 23:59:07 (490040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:04:09 (488464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:04:11 (488464): No heartbeat from core client for 30 sec - exiting 00:12:33 (490360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:35 (490360): No heartbeat from core client for 30 sec - exiting 00:14:11 (490468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:39:13 (491388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:39:15 (491388): No heartbeat from core client for 30 sec - exiting 00:39:16 (491388): No heartbeat from core client for 30 sec - exiting 00:39:17 (491388): No heartbeat from core client for 30 sec - exiting 00:49:15 (490468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:49:18 (490468): No heartbeat from core client for 30 sec - exiting 00:49:19 (490468): No heartbeat from core client for 30 sec - exiting 00:49:20 (490468): No heartbeat from core client for 30 sec - exiting 00:54:14 (491588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:14 (492964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:15 (492964): No heartbeat from core client for 30 sec - exiting 01:04:14 (493212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:04:15 (493212): No heartbeat from core client for 30 sec - exiting 01:09:15 (489580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:14:16 (493492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:14:17 (493492): No heartbeat from core client for 30 sec - exiting 01:19:16 (493920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:24:18 (494496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:24:19 (494496): No heartbeat from core client for 30 sec - exiting 01:25:38 (494192): No heartbeat from core client for 30 sec - exiting 01:25:39 (494192): No heartbeat from core client for 30 sec - exiting 01:25:40 (494192): No heartbeat from core client for 30 sec - exiting 01:25:41 (494192): No heartbeat from core client for 30 sec - exiting 01:25:42 (494192): No heartbeat from core client for 30 sec - exiting 01:25:43 (494192): No heartbeat from core client for 30 sec - exiting 01:25:44 (494192): No heartbeat from core client for 30 sec - exiting 01:25:45 (494192): No heartbeat from core client for 30 sec - exiting 01:25:46 (494192): No heartbeat from core client for 30 sec - exiting 01:25:47 (494192): No heartbeat from core client for 30 sec - exiting 01:25:48 (494192): No heartbeat from core client for 30 sec - exiting 01:25:49 (494192): No heartbeat from core client for 30 sec - exiting 01:25:50 (494192): No heartbeat from core client for 30 sec - exiting 01:25:51 (494192): No heartbeat from core client for 30 sec - exiting 01:25:52 (494192): No heartbeat from core client for 30 sec - exiting 01:25:53 (494192): No heartbeat from core client for 30 sec - exiting 01:25:54 (494192): No heartbeat from core client for 30 sec - exiting 01:25:55 (494192): No heartbeat from core client for 30 sec - exiting 01:25:56 (494192): No heartbeat from core client for 30 sec - exiting 01:25:57 (494192): No heartbeat from core client for 30 sec - exiting 01:25:58 (494192): No heartbeat from core client for 30 sec - exiting 01:25:59 (494192): No heartbeat from core client for 30 sec - exiting 01:26:00 (494192): No heartbeat from core client for 30 sec - exiting 01:26:01 (494192): No heartbeat from core client for 30 sec - exiting 01:26:02 (494192): No heartbeat from core client for 30 sec - exiting 01:26:03 (494192): No heartbeat from core client for 30 sec - exiting 01:26:04 (494192): No heartbeat from core client for 30 sec - exiting 01:26:05 (494192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:29:17 (493060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:34:21 (491152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:49:24 (493780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:49:26 (493780): No heartbeat from core client for 30 sec - exiting 01:49:27 (493780): No heartbeat from core client for 30 sec - exiting 01:54:26 (494300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:55:30 (493200): No heartbeat from core client for 30 sec - exiting 01:55:31 (493200): No heartbeat from core client for 30 sec - exiting 01:55:32 (493200): No heartbeat from core client for 30 sec - exiting 01:55:33 (493200): No heartbeat from core client for 30 sec - exiting 01:55:34 (493200): No heartbeat from core client for 30 sec - exiting 01:55:35 (493200): No heartbeat from core client for 30 sec - exiting 01:55:36 (493200): No heartbeat from core client for 30 sec - exiting 01:55:37 (493200): No heartbeat from core client for 30 sec - exiting 01:55:38 (493200): No heartbeat from core client for 30 sec - exiting 01:55:39 (493200): No heartbeat from core client for 30 sec - exiting 01:55:40 (493200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:59:34 (493008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:59:35 (493008): No heartbeat from core client for 30 sec - exiting 02:04:35 (492568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:09:35 (494852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:09:36 (494852): No heartbeat from core client for 30 sec - exiting 02:14:35 (493608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:19:36 (493920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:19:38 (493920): No heartbeat from core client for 30 sec - exiting 02:19:39 (493920): No heartbeat from core client for 30 sec - exiting 02:29:36 (495688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:29:37 (495688): No heartbeat from core client for 30 sec - exiting 02:34:37 (496560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:34:46 (496560): No heartbeat from core client for 30 sec - exiting 02:39:38 (493452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:39 (493452): No heartbeat from core client for 30 sec - exiting 02:49:39 (495452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:49:41 (495452): No heartbeat from core client for 30 sec - exiting 02:49:42 (495452): No heartbeat from core client for 30 sec - exiting 02:54:41 (489320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:56:35 (497168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:59:42 (497644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:04:42 (497060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:04:44 (497060): No heartbeat from core client for 30 sec - exiting 03:09:42 (495852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:09:43 (495852): No heartbeat from core client for 30 sec - exiting 03:14:42 (494400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:19:59 (496896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:00 (496896): No heartbeat from core client for 30 sec - exiting 03:25:02 (497840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:00 (498272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:00 (494172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:02 (494172): No heartbeat from core client for 30 sec - exiting 03:45:03 (494172): No heartbeat from core client for 30 sec - exiting 03:50:02 (497120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:05:01 (498524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:05:02 (498524): No heartbeat from core client for 30 sec - exiting 04:05:58 (490548): No heartbeat from core client for 30 sec - exiting 04:06:00 (490548): No heartbeat from core client for 30 sec - exiting 04:06:01 (490548): No heartbeat from core client for 30 sec - exiting 04:06:02 (490548): No heartbeat from core client for 30 sec - exiting 04:06:03 (490548): No heartbeat from core client for 30 sec - exiting 04:06:04 (490548): No heartbeat from core client for 30 sec - exiting 04:06:05 (490548): No heartbeat from core client for 30 sec - exiting 04:06:06 (490548): No heartbeat from core client for 30 sec - exiting 04:06:07 (490548): No heartbeat from core client for 30 sec - exiting 04:06:08 (490548): No heartbeat from core client for 30 sec - exiting 04:06:09 (490548): No heartbeat from core client for 30 sec - exiting 04:06:10 (490548): No heartbeat from core client for 30 sec - exiting 04:06:11 (490548): No heartbeat from core client for 30 sec - exiting 04:06:12 (490548): No heartbeat from core client for 30 sec - exiting 04:06:13 (490548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:06:14 (490548): No heartbeat from core client for 30 sec - exiting 04:06:15 (490548): No heartbeat from core client for 30 sec - exiting 04:10:03 (498736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:39:46 (5284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4192, iMonCtr=1 Model crash detected, will try to restart... 03:34:45 (9184): No heartbeat from core client for 30 sec - exiting 03:34:48 (9184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7136, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7136, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7136, iMonCtr=1 Model crash detected, will try to restart... 18:29:32 (7136): No heartbeat from core client for 30 sec - exiting 18:29:33 (7136): No heartbeat from core client for 30 sec - exiting 18:29:34 (7136): No heartbeat from core client for 30 sec - exiting 18:29:35 (7136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8784, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8784, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8784, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... 22:18:58 (61820): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 22:18:59 (61820): No heartbeat from core client for 30 sec - exiting 22:19:00 (61820): No heartbeat from core client for 30 sec - exiting 22:19:01 (61820): No heartbeat from core client for 30 sec - exiting 22:19:02 (61820): No heartbeat from core client for 30 sec - exiting 22:19:03 (61820): No heartbeat from core client for 30 sec - exiting 22:19:04 (61820): No heartbeat from core client for 30 sec - exiting 22:19:05 (61820): No heartbeat from core client for 30 sec - exiting 22:19:06 (61820): No heartbeat from core client for 30 sec - exiting 22:19:07 (61820): No heartbeat from core client for 30 sec - exiting 22:19:08 (61820): No heartbeat from core client for 30 sec - exiting 13:14:18 (76108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:48:36 (110528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:48:38 (110528): No heartbeat from core client for 30 sec - exiting 15:48:39 (110528): No heartbeat from core client for 30 sec - exiting 15:48:40 (110528): No heartbeat from core client for 30 sec - exiting 15:48:41 (110528): No heartbeat from core client for 30 sec - exiting 15:48:42 (110528): No heartbeat from core client for 30 sec - exiting 15:53:37 (175968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:24:15 (173724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:38 (180492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=180900, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=180900, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=180900, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=180900, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=180900, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=180900, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=158836, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=158836, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=158836, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=158836, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=158836, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=158836, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Sep 2013 17:11:45	1137800	15924208	hadcm3n_n26h_1920_40_008371713_1	77,760	218,389	2.8085
20 Aug 2013 19:56:31	1137800	15924208	hadcm3n_n26h_1920_40_008371713_1	51,840	82,489	1.5912
20 Aug 2013 07:32:53	1137800	15924208	hadcm3n_n26h_1920_40_008371713_1	25,920	40,825	1.5750