Task 15815525

Name	hadcm3n_o4pk_1940_40_008383526_2
Workunit	8534385
Created	1 Jun 2013, 10:48:32 UTC
Sent	1 Jun 2013, 11:11:17 UTC
Report deadline	31 Aug 2013, 18:38:28 UTC
Received	22 Jun 2013, 22:54:33 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1237173
Run time	3 days 4 hours 58 min 21 sec
CPU time	2 days 14 hours 7 min 46 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	3.35 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 00:45:16 (27856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:41:57 (25624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:42:02 (25624): No heartbeat from core client for 30 sec - exiting 06:57:45 (30076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:26:56 (29200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:06 (24896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:11 (24896): No heartbeat from core client for 30 sec - exiting 07:52:12 (24896): No heartbeat from core client for 30 sec - exiting 07:52:13 (24896): No heartbeat from core client for 30 sec - exiting 07:52:14 (24896): No heartbeat from core client for 30 sec - exiting 07:52:15 (24896): No heartbeat from core client for 30 sec - exiting 07:52:16 (24896): No heartbeat from core client for 30 sec - exiting 07:52:17 (24896): No heartbeat from core client for 30 sec - exiting 07:52:18 (24896): No heartbeat from core client for 30 sec - exiting 07:52:19 (24896): No heartbeat from core client for 30 sec - exiting 07:52:20 (24896): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 08:09:15 (30728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:16 (30728): No heartbeat from core client for 30 sec - exiting 10:06:42 (29364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:43 (29364): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 10:10:46 (31264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:10:47 (31264): No heartbeat from core client for 30 sec - exiting 10:10:48 (31264): No heartbeat from core client for 30 sec - exiting 10:10:49 (31264): No heartbeat from core client for 30 sec - exiting 10:10:50 (31264): No heartbeat from core client for 30 sec - exiting 10:10:51 (31264): No heartbeat from core client for 30 sec - exiting 10:10:52 (31264): No heartbeat from core client for 30 sec - exiting 10:10:53 (31264): No heartbeat from core client for 30 sec - exiting 10:10:54 (31264): No heartbeat from core client for 30 sec - exiting 10:10:55 (31264): No heartbeat from core client for 30 sec - exiting 10:10:56 (31264): No heartbeat from core client for 30 sec - exiting 10:48:36 (17636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:32:57 (26632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:12:39 (30760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:01 (32312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:20:54 (32048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:56:34 (6384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:56:36 (6384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 22:07:36 (10904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:07:37 (10904): No heartbeat from core client for 30 sec - exiting 22:07:38 (10904): No heartbeat from core client for 30 sec - exiting 22:07:39 (10904): No heartbeat from core client for 30 sec - exiting 22:07:40 (10904): No heartbeat from core client for 30 sec - exiting 22:07:41 (10904): No heartbeat from core client for 30 sec - exiting 22:07:42 (10904): No heartbeat from core client for 30 sec - exiting 22:07:43 (10904): No heartbeat from core client for 30 sec - exiting 22:07:57 (10456): Can't acquire lockfile (32) - waiting 35s 06:36:50 (10456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:36:56 (10456): No heartbeat from core client for 30 sec - exiting 06:41:16 (8072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:21 (1268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:51 (12496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:46:59 (4888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:03 (4888): No heartbeat from core client for 30 sec - exiting 09:01:09 (9412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:14 (10684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:19:53 (9372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:07:02 (9428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:17 (12352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2852, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2852, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2852, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2852, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2852, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2852, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish 12:36:00 (13612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:57:57 (13332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:57:58 (13332): No heartbeat from core client for 30 sec - exiting 12:57:59 (13332): No heartbeat from core client for 30 sec - exiting 12:58:00 (13332): No heartbeat from core client for 30 sec - exiting 12:58:01 (13332): No heartbeat from core client for 30 sec - exiting 12:58:02 (13332): No heartbeat from core client for 30 sec - exiting 12:58:03 (13332): No heartbeat from core client for 30 sec - exiting 12:58:04 (13332): No heartbeat from core client for 30 sec - exiting 12:58:05 (13332): No heartbeat from core client for 30 sec - exiting 12:58:06 (13332): No heartbeat from core client for 30 sec - exiting 12:58:07 (13332): No heartbeat from core client for 30 sec - exiting 13:17:45 (13900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:43:41 (12372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:43:42 (12372): No heartbeat from core client for 30 sec - exiting 15:13:44 (13820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:43:05 (3460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:43:26 (3460): No heartbeat from core client for 30 sec - exiting 16:16:41 (15056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:46 (14596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:48 (14596): No heartbeat from core client for 30 sec - exiting 17:53:38 (14936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:57 (11572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:00 (11572): No heartbeat from core client for 30 sec - exiting 18:31:01 (11572): No heartbeat from core client for 30 sec - exiting 18:31:02 (11572): No heartbeat from core client for 30 sec - exiting 18:31:03 (11572): No heartbeat from core client for 30 sec - exiting 18:31:04 (11572): No heartbeat from core client for 30 sec - exiting 19:27:03 (8900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:13:32 (15812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:44:56 (19312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:58:23 (17920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:48:03 (19488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:03:00 (5860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:10:18 (20392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:10:24 (20392): No heartbeat from core client for 30 sec - exiting 00:10:25 (20392): No heartbeat from core client for 30 sec - exiting 00:10:26 (20392): No heartbeat from core client for 30 sec - exiting 00:10:27 (20392): No heartbeat from core client for 30 sec - exiting 00:10:28 (20392): No heartbeat from core client for 30 sec - exiting 00:10:29 (20392): No heartbeat from core client for 30 sec - exiting 00:10:30 (20392): No heartbeat from core client for 30 sec - exiting 00:10:31 (20392): No heartbeat from core client for 30 sec - exiting 00:10:32 (20392): No heartbeat from core client for 30 sec - exiting 00:30:16 (19288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:18:17 (19160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:18:22 (19160): No heartbeat from core client for 30 sec - exiting 01:18:23 (19160): No heartbeat from core client for 30 sec - exiting 01:18:24 (19160): No heartbeat from core client for 30 sec - exiting 01:18:25 (19160): No heartbeat from core client for 30 sec - exiting 01:18:26 (19160): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 05:09:47 (20468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:16:21 (18324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:16:26 (18324): No heartbeat from core client for 30 sec - exiting 11:17:26 (18064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:17:42 (18064): No heartbeat from core client for 30 sec - exiting 11:17:43 (18064): No heartbeat from core client for 30 sec - exiting 11:17:44 (18064): No heartbeat from core client for 30 sec - exiting 11:17:45 (18064): No heartbeat from core client for 30 sec - exiting 11:17:46 (18064): No heartbeat from core client for 30 sec - exiting 11:17:47 (18064): No heartbeat from core client for 30 sec - exiting 11:17:48 (18064): No heartbeat from core client for 30 sec - exiting 11:17:49 (18064): No heartbeat from core client for 30 sec - exiting 11:17:50 (18064): No heartbeat from core client for 30 sec - exiting 11:17:51 (18064): No heartbeat from core client for 30 sec - exiting 11:37:43 (19272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:36:00 (21492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:51:24 (20696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:51:29 (20696): No heartbeat from core client for 30 sec - exiting 13:51:30 (20696): No heartbeat from core client for 30 sec - exiting 13:51:31 (20696): No heartbeat from core client for 30 sec - exiting 14:44:53 (18892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:11:45 (20892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:11:49 (20892): No heartbeat from core client for 30 sec - exiting 17:11:50 (20892): No heartbeat from core client for 30 sec - exiting 17:11:51 (20892): No heartbeat from core client for 30 sec - exiting 17:11:52 (20892): No heartbeat from core client for 30 sec - exiting 17:11:53 (20892): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 17:20:28 (22208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:29 (22208): No heartbeat from core client for 30 sec - exiting 17:20:30 (22208): No heartbeat from core client for 30 sec - exiting 17:20:31 (22208): No heartbeat from core client for 30 sec - exiting 17:20:32 (22208): No heartbeat from core client for 30 sec - exiting 17:20:33 (22208): No heartbeat from core client for 30 sec - exiting 17:20:34 (22208): No heartbeat from core client for 30 sec - exiting 17:20:35 (22208): No heartbeat from core client for 30 sec - exiting 17:20:36 (22208): No heartbeat from core client for 30 sec - exiting 17:20:37 (22208): No heartbeat from core client for 30 sec - exiting 17:20:38 (22208): No heartbeat from core client for 30 sec - exiting 18:18:23 (22112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:29 (22112): No heartbeat from core client for 30 sec - exiting 18:18:30 (22112): No heartbeat from core client for 30 sec - exiting 18:18:31 (22112): No heartbeat from core client for 30 sec - exiting 18:18:32 (22112): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6464, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6464, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6464, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6464, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6464, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6464, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Jun 2013 19:44:10	1237173	15815525	hadcm3n_o4pk_1940_40_008383526_2	181,440	223,688	1.2328
22 Jun 2013 14:23:43	1237173	15815525	hadcm3n_o4pk_1940_40_008383526_2	155,520	205,451	1.3211
22 Jun 2013 09:01:31	1237173	15815525	hadcm3n_o4pk_1940_40_008383526_2	129,600	187,249	1.4448
22 Jun 2013 03:33:02	1237173	15815525	hadcm3n_o4pk_1940_40_008383526_2	103,680	169,005	1.6301
21 Jun 2013 22:13:16	1237173	15815525	hadcm3n_o4pk_1940_40_008383526_2	77,760	150,773	1.9390
19 Jun 2013 23:43:33	1237173	15815525	hadcm3n_o4pk_1940_40_008383526_2	51,840	54,421	1.0498
19 Jun 2013 15:49:32	1237173	15815525	hadcm3n_o4pk_1940_40_008383526_2	25,920	29,150	1.1246