Task 15822208

Name	hadcm3n_o6lt_1980_40_008386217_1
Workunit	8537076
Created	3 Jun 2013, 7:04:10 UTC
Sent	3 Jun 2013, 7:18:47 UTC
Report deadline	2 Sep 2013, 14:45:58 UTC
Received	21 Jun 2013, 1:24:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1237173
Run time	1 days 7 hours 51 min 50 sec
CPU time	22 hours 3 min 55 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	3.35 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 04:41:57 (30632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:42:02 (30632): No heartbeat from core client for 30 sec - exiting 06:57:45 (29976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:26:56 (31152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:06 (30768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:11 (30768): No heartbeat from core client for 30 sec - exiting 07:52:12 (30768): No heartbeat from core client for 30 sec - exiting 07:52:13 (30768): No heartbeat from core client for 30 sec - exiting 07:52:14 (30768): No heartbeat from core client for 30 sec - exiting 07:52:15 (30768): No heartbeat from core client for 30 sec - exiting 07:52:16 (30768): No heartbeat from core client for 30 sec - exiting 07:52:17 (30768): No heartbeat from core client for 30 sec - exiting 07:52:18 (30768): No heartbeat from core client for 30 sec - exiting 07:52:19 (30768): No heartbeat from core client for 30 sec - exiting 07:52:20 (30768): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 08:09:15 (29936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:41 (31460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:43 (31460): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 10:10:45 (31372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:10:46 (31372): No heartbeat from core client for 30 sec - exiting 10:10:47 (31372): No heartbeat from core client for 30 sec - exiting 10:10:48 (31372): No heartbeat from core client for 30 sec - exiting 10:10:49 (31372): No heartbeat from core client for 30 sec - exiting 10:10:50 (31372): No heartbeat from core client for 30 sec - exiting 10:10:51 (31372): No heartbeat from core client for 30 sec - exiting 10:10:52 (31372): No heartbeat from core client for 30 sec - exiting 10:10:53 (31372): No heartbeat from core client for 30 sec - exiting 10:10:54 (31372): No heartbeat from core client for 30 sec - exiting 10:10:55 (31372): No heartbeat from core client for 30 sec - exiting 10:48:36 (32560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:32:58 (32864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:12:39 (30040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:01 (31700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:04 (31700): No heartbeat from core client for 30 sec - exiting 15:20:54 (27052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:56:34 (6460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:56:35 (6460): No heartbeat from core client for 30 sec - exiting 21:56:36 (6460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 22:31:02 (10876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:36:53 (6356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:36:56 (6356): No heartbeat from core client for 30 sec - exiting 06:41:13 (5728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:41:17 (5728): No heartbeat from core client for 30 sec - exiting 06:41:18 (5728): No heartbeat from core client for 30 sec - exiting 06:50:57 (4248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 07:09:13 (13200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:14 (13200): No heartbeat from core client for 30 sec - exiting 07:09:15 (13200): No heartbeat from core client for 30 sec - exiting 07:09:16 (13200): No heartbeat from core client for 30 sec - exiting 07:09:17 (13200): No heartbeat from core client for 30 sec - exiting 07:09:18 (13200): No heartbeat from core client for 30 sec - exiting 07:09:19 (13200): No heartbeat from core client for 30 sec - exiting 07:31:46 (10308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:55 (10308): No heartbeat from core client for 30 sec - exiting 07:31:57 (10308): No heartbeat from core client for 30 sec - exiting 07:31:58 (10308): No heartbeat from core client for 30 sec - exiting 07:31:59 (10308): No heartbeat from core client for 30 sec - exiting 07:32:00 (10308): No heartbeat from core client for 30 sec - exiting 08:46:59 (7024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:03 (7024): No heartbeat from core client for 30 sec - exiting 08:47:04 (7024): No heartbeat from core client for 30 sec - exiting 08:47:05 (7024): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 09:01:08 (11452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:01:09 (11452): No heartbeat from core client for 30 sec - exiting 09:13:13 (11884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:19:54 (11856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:20:02 (11856): No heartbeat from core client for 30 sec - exiting 09:20:03 (11856): No heartbeat from core client for 30 sec - exiting 09:20:04 (11856): No heartbeat from core client for 30 sec - exiting 09:20:05 (11856): No heartbeat from core client for 30 sec - exiting 09:20:06 (11856): No heartbeat from core client for 30 sec - exiting 09:20:07 (11856): No heartbeat from core client for 30 sec - exiting 09:20:08 (11856): No heartbeat from core client for 30 sec - exiting 09:20:09 (11856): No heartbeat from core client for 30 sec - exiting 09:20:10 (11856): No heartbeat from core client for 30 sec - exiting 09:20:11 (11856): No heartbeat from core client for 30 sec - exiting 11:07:02 (11680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:17 (12716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:21 (12716): No heartbeat from core client for 30 sec - exiting 11:50:22 (12716): No heartbeat from core client for 30 sec - exiting 11:50:23 (12716): No heartbeat from core client for 30 sec - exiting 11:50:24 (12716): No heartbeat from core client for 30 sec - exiting 11:50:25 (12716): No heartbeat from core client for 30 sec - exiting 11:50:26 (12716): No heartbeat from core client for 30 sec - exiting 11:50:27 (12716): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 12:28:29 (13988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:36:00 (13672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:57:57 (6300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:57:58 (6300): No heartbeat from core client for 30 sec - exiting 12:57:59 (6300): No heartbeat from core client for 30 sec - exiting 12:58:00 (6300): No heartbeat from core client for 30 sec - exiting 12:58:01 (6300): No heartbeat from core client for 30 sec - exiting 12:58:02 (6300): No heartbeat from core client for 30 sec - exiting 12:58:03 (6300): No heartbeat from core client for 30 sec - exiting 12:58:04 (6300): No heartbeat from core client for 30 sec - exiting 12:58:05 (6300): No heartbeat from core client for 30 sec - exiting 12:58:06 (6300): No heartbeat from core client for 30 sec - exiting 12:58:07 (6300): No heartbeat from core client for 30 sec - exiting 13:17:46 (8948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 14:43:42 (13508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:13:44 (15156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:43:05 (10032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:43:26 (10032): No heartbeat from core client for 30 sec - exiting 16:16:40 (12012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:46 (14412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:53:38 (14784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:57 (8516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:27:03 (4232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17000, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17000, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17000, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17000, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17000, iMonCtr=1 Model crash detected, will try to restart... 21:13:32 (17000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:13:33 (17000): No heartbeat from core client for 30 sec - exiting 21:13:34 (17000): No heartbeat from core client for 30 sec - exiting 21:13:35 (17000): No heartbeat from core client for 30 sec - exiting 21:13:36 (17000): No heartbeat from core client for 30 sec - exiting 21:13:37 (17000): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19568, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Jun 2013 10:28:41	1237173	15822208	hadcm3n_o6lt_1980_40_008386217_1	51,840	54,432	1.0500
19 Jun 2013 19:24:08	1237173	15822208	hadcm3n_o6lt_1980_40_008386217_1	25,920	28,053	1.0823