Task 16585673

Name	hadcm3n_88uf_1980_40_008720882_0
Workunit	8866860
Created	23 Apr 2014, 12:28:39 UTC
Sent	5 May 2014, 15:02:56 UTC
Report deadline	4 Aug 2014, 22:30:07 UTC
Received	26 May 2014, 4:13:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1169946
Run time	3 days 22 hours 36 min 37 sec
CPU time	3 days 13 hours 42 min 24 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	3.25 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 07:26:11 (6164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:16:26 (6524): No heartbeat from core client for 30 sec - exiting 21:16:27 (6524): No heartbeat from core client for 30 sec - exiting 21:16:28 (6524): No heartbeat from core client for 30 sec - exiting 21:16:29 (6524): No heartbeat from core client for 30 sec - exiting 21:16:30 (6524): No heartbeat from core client for 30 sec - exiting 21:16:31 (6524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:54:41 (6644): No heartbeat from core client for 30 sec - exiting 08:54:42 (6644): No heartbeat from core client for 30 sec - exiting 08:54:43 (6644): No heartbeat from core client for 30 sec - exiting 08:54:44 (6644): No heartbeat from core client for 30 sec - exiting 08:54:45 (6644): No heartbeat from core client for 30 sec - exiting 08:54:46 (6644): No heartbeat from core client for 30 sec - exiting 08:54:47 (6644): No heartbeat from core client for 30 sec - exiting 08:54:48 (6644): No heartbeat from core client for 30 sec - exiting 08:54:49 (6644): No heartbeat from core client for 30 sec - exiting 08:54:50 (6644): No heartbeat from core client for 30 sec - exiting 08:54:51 (6644): No heartbeat from core client for 30 sec - exiting 08:54:52 (6644): No heartbeat from core client for 30 sec - exiting 08:54:53 (6644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:25:41 (6968): No heartbeat from core client for 30 sec - exiting 22:25:42 (6968): No heartbeat from core client for 30 sec - exiting 22:25:43 (6968): No heartbeat from core client for 30 sec - exiting 22:25:44 (6968): No heartbeat from core client for 30 sec - exiting 22:25:45 (6968): No heartbeat from core client for 30 sec - exiting 22:25:46 (6968): No heartbeat from core client for 30 sec - exiting 22:25:47 (6968): No heartbeat from core client for 30 sec - exiting 22:25:48 (6968): No heartbeat from core client for 30 sec - exiting 22:25:49 (6968): No heartbeat from core client for 30 sec - exiting 22:25:50 (6968): No heartbeat from core client for 30 sec - exiting 22:25:51 (6968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:25:13 (4244): No heartbeat from core client for 30 sec - exiting 09:25:15 (4244): No heartbeat from core client for 30 sec - exiting 09:25:16 (4244): No heartbeat from core client for 30 sec - exiting 09:25:17 (4244): No heartbeat from core client for 30 sec - exiting 09:25:18 (4244): No heartbeat from core client for 30 sec - exiting 09:25:19 (4244): No heartbeat from core client for 30 sec - exiting 09:25:20 (4244): No heartbeat from core client for 30 sec - exiting 09:25:21 (4244): No heartbeat from core client for 30 sec - exiting 09:25:22 (4244): No heartbeat from core client for 30 sec - exiting 09:25:23 (4244): No heartbeat from core client for 30 sec - exiting 09:25:24 (4244): No heartbeat from core client for 30 sec - exiting 09:25:25 (4244): No heartbeat from core client for 30 sec - exiting 09:25:26 (4244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:25:58 (6568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:09:15 (6564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:45:20 (3904): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 18:15:50 (4112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:15:51 (4112): No heartbeat from core client for 30 sec - exiting 18:15:52 (4112): No heartbeat from core client for 30 sec - exiting 18:15:53 (4112): No heartbeat from core client for 30 sec - exiting 18:15:54 (4112): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4932, iMonCtr=1 Model crash detected, will try to restart... 07:30:22 (4512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7008, iMonCtr=1 Model crash detected, will try to restart... 07:13:43 (6004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:43 (4928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6944, iMonCtr=1 Model crash detected, will try to restart... 10:01:57 (5368): No heartbeat from core client for 30 sec - exiting 10:01:58 (5368): No heartbeat from core client for 30 sec - exiting 10:01:59 (5368): No heartbeat from core client for 30 sec - exiting 10:02:00 (5368): No heartbeat from core client for 30 sec - exiting 10:02:01 (5368): No heartbeat from core client for 30 sec - exiting 10:02:02 (5368): No heartbeat from core client for 30 sec - exiting 10:02:03 (5368): No heartbeat from core client for 30 sec - exiting 10:02:04 (5368): No heartbeat from core client for 30 sec - exiting 10:02:05 (5368): No heartbeat from core client for 30 sec - exiting 10:02:06 (5368): No heartbeat from core client for 30 sec - exiting 10:02:07 (5368): No heartbeat from core client for 30 sec - exiting 10:02:08 (5368): No heartbeat from core client for 30 sec - exiting 10:02:09 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:09:21 (4792): No heartbeat from core client for 30 sec - exiting 21:09:23 (4792): No heartbeat from core client for 30 sec - exiting 21:09:24 (4792): No heartbeat from core client for 30 sec - exiting 21:09:25 (4792): No heartbeat from core client for 30 sec - exiting 21:09:26 (4792): No heartbeat from core client for 30 sec - exiting 21:09:27 (4792): No heartbeat from core client for 30 sec - exiting 21:09:28 (4792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:24 (7084): No heartbeat from core client for 30 sec - exiting 08:13:25 (7084): No heartbeat from core client for 30 sec - exiting 08:13:26 (7084): No heartbeat from core client for 30 sec - exiting 08:13:27 (7084): No heartbeat from core client for 30 sec - exiting 08:13:28 (7084): No heartbeat from core client for 30 sec - exiting 08:13:29 (7084): No heartbeat from core client for 30 sec - exiting 08:13:30 (7084): No heartbeat from core client for 30 sec - exiting 08:13:31 (7084): No heartbeat from core client for 30 sec - exiting 08:13:32 (7084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:38:23 (5476): No heartbeat from core client for 30 sec - exiting 10:38:24 (5476): No heartbeat from core client for 30 sec - exiting 10:38:25 (5476): No heartbeat from core client for 30 sec - exiting 10:38:26 (5476): No heartbeat from core client for 30 sec - exiting 10:38:27 (5476): No heartbeat from core client for 30 sec - exiting 10:38:28 (5476): No heartbeat from core client for 30 sec - exiting 10:38:29 (5476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:54:49 (6952): No heartbeat from core client for 30 sec - exiting 22:54:50 (6952): No heartbeat from core client for 30 sec - exiting 22:54:51 (6952): No heartbeat from core client for 30 sec - exiting 22:54:52 (6952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:27:23 (2848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:32:53 (6056): No heartbeat from core client for 30 sec - exiting 19:32:54 (6056): No heartbeat from core client for 30 sec - exiting 19:32:55 (6056): No heartbeat from core client for 30 sec - exiting 19:32:56 (6056): No heartbeat from core client for 30 sec - exiting 19:32:57 (6056): No heartbeat from core client for 30 sec - exiting 19:32:58 (6056): No heartbeat from core client for 30 sec - exiting 19:32:59 (6056): No heartbeat from core client for 30 sec - exiting 19:33:00 (6056): No heartbeat from core client for 30 sec - exiting 19:33:01 (6056): No heartbeat from core client for 30 sec - exiting 19:33:02 (6056): No heartbeat from core client for 30 sec - exiting 19:33:03 (6056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:07:21 (6772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:46:02 (6252): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3628, iMonCtr=1 Model crash detected, will try to restart... 07:20:20 (1940): No heartbeat from core client for 30 sec - exiting 07:20:21 (1940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:42:02 (6264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6856, iMonCtr=1 Model crash detected, will try to restart... 10:15:19 (6080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:08:56 (3720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:03:09 (3136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:38:54 (6468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:08 (5756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:40 (6972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:03:23 (212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:44:24 (5812): No heartbeat from core client for 30 sec - exiting 21:44:25 (5812): No heartbeat from core client for 30 sec - exiting 21:44:26 (5812): No heartbeat from core client for 30 sec - exiting 21:44:27 (5812): No heartbeat from core client for 30 sec - exiting 21:44:28 (5812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:24:35 (5868): No heartbeat from core client for 30 sec - exiting 07:24:36 (5868): No heartbeat from core client for 30 sec - exiting 07:24:37 (5868): No heartbeat from core client for 30 sec - exiting 07:24:38 (5868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:30:05 (6008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:48 (7112): No heartbeat from core client for 30 sec - exiting 20:05:49 (7112): No heartbeat from core client for 30 sec - exiting 20:05:50 (7112): No heartbeat from core client for 30 sec - exiting 20:05:51 (7112): No heartbeat from core client for 30 sec - exiting 20:05:52 (7112): No heartbeat from core client for 30 sec - exiting 20:05:53 (7112): No heartbeat from core client for 30 sec - exiting 20:05:54 (7112): No heartbeat from core client for 30 sec - exiting 20:05:55 (7112): No heartbeat from core client for 30 sec - exiting 20:05:56 (7112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4152, iMonCtr=1 Model crash detected, will try to restart... 20:23:08 (6476): No heartbeat from core client for 30 sec - exiting 20:23:09 (6476): No heartbeat from core client for 30 sec - exiting 20:23:10 (6476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:29:50 (7152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:22:32 (5752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:23:25 (3520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:38:30 (6812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:49:33 (3124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:16:35 (6076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:26:23 (6408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7028, iMonCtr=1 Model crash detected, will try to restart... 11:33:44 (3752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:11:36 (6800): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5536, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5536, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5536, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5536, iMonCtr=1 Model crash detected, will try to restart... 20:11:11 (5536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1980, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1980, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 May 2014 23:11:13	1169946	16585673	hadcm3n_88uf_1980_40_008720882_0	233,280	305,753	1.3107
25 May 2014 03:25:32	1169946	16585673	hadcm3n_88uf_1980_40_008720882_0	207,360	272,967	1.3164
22 May 2014 16:56:12	1169946	16585673	hadcm3n_88uf_1980_40_008720882_0	181,440	238,542	1.3147
20 May 2014 04:35:53	1169946	16585673	hadcm3n_88uf_1980_40_008720882_0	155,520	205,881	1.3238
17 May 2014 06:01:00	1169946	16585673	hadcm3n_88uf_1980_40_008720882_0	129,600	171,977	1.3270
15 May 2014 09:55:22	1169946	16585673	hadcm3n_88uf_1980_40_008720882_0	103,680	136,212	1.3138
11 May 2014 01:54:51	1169946	16585673	hadcm3n_88uf_1980_40_008720882_0	77,760	102,912	1.3235
10 May 2014 15:20:31	1169946	16585673	hadcm3n_88uf_1980_40_008720882_0	51,840	68,543	1.3222
08 May 2014 22:13:22	1169946	16585673	hadcm3n_88uf_1980_40_008720882_0	25,920	34,230	1.3206