Task 15465274

Name	hadcm3n_o5ea_2100_40_008254521_0
Workunit	8409645
Created	28 Nov 2012, 13:32:38 UTC
Sent	28 Nov 2012, 13:37:18 UTC
Report deadline	27 Feb 2013, 21:04:29 UTC
Received	24 Dec 2012, 15:29:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	270720
Run time	9 days 6 hours 15 min 12 sec
CPU time	9 days 6 hours 15 min 12 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	1.25 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>5.2.7</core_client_version> <message>The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 20:54:21 (3480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:05:53 (4080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:04:35 (14116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:07:25 (3960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:41:02 (4036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:03:04 (10492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:12:15 (508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:11:06 (11772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:10:01 (21668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:08:54 (32216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:43:14 (1128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:07:42 (1644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:06:28 (24700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:21:41 (3840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:29:27 (1268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:28:22 (14928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:34:09 (3404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:25:05 (1408): No heartbeat from core client for 30 sec - exiting 17:25:07 (1408): No heartbeat from core client for 30 sec - exiting 17:25:08 (1408): No heartbeat from core client for 30 sec - exiting 17:25:09 (1408): No heartbeat from core client for 30 sec - exiting 17:25:10 (1408): No heartbeat from core client for 30 sec - exiting 17:25:11 (1408): No heartbeat from core client for 30 sec - exiting 17:25:12 (1408): No heartbeat from core client for 30 sec - exiting 17:25:13 (1408): No heartbeat from core client for 30 sec - exiting 17:25:14 (1408): No heartbeat from core client for 30 sec - exiting 17:25:15 (1408): No heartbeat from core client for 30 sec - exiting 17:25:17 (1408): No heartbeat from core client for 30 sec - exiting 17:25:18 (1408): No heartbeat from core client for 30 sec - exiting 17:25:19 (1408): No heartbeat from core client for 30 sec - exiting 17:25:20 (1408): No heartbeat from core client for 30 sec - exiting 17:25:21 (1408): No heartbeat from core client for 30 sec - exiting 17:25:22 (1408): No heartbeat from core client for 30 sec - exiting 17:25:23 (1408): No heartbeat from core client for 30 sec - exiting 17:25:24 (1408): No heartbeat from core client for 30 sec - exiting 17:25:26 (1408): No heartbeat from core client for 30 sec - exiting 17:25:27 (1408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:25:28 (1408): No heartbeat from core client for 30 sec - exiting 20:18:13 (940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:31:01 (3676): No heartbeat from core client for 30 sec - exiting 08:31:02 (3676): No heartbeat from core client for 30 sec - exiting 08:31:03 (3676): No heartbeat from core client for 30 sec - exiting 08:31:04 (3676): No heartbeat from core client for 30 sec - exiting 08:31:07 (3676): No heartbeat from core client for 30 sec - exiting 08:31:08 (3676): No heartbeat from core client for 30 sec - exiting 08:31:09 (3676): No heartbeat from core client for 30 sec - exiting 08:31:11 (3676): No heartbeat from core client for 30 sec - exiting 08:31:12 (3676): No heartbeat from core client for 30 sec - exiting 08:31:13 (3676): No heartbeat from core client for 30 sec - exiting 08:31:14 (3676): No heartbeat from core client for 30 sec - exiting 08:31:15 (3676): No heartbeat from core client for 30 sec - exiting 08:31:16 (3676): No heartbeat from core client for 30 sec - exiting 08:31:17 (3676): No heartbeat from core client for 30 sec - exiting 08:31:18 (3676): No heartbeat from core client for 30 sec - exiting 08:31:20 (3676): No heartbeat from core client for 30 sec - exiting 08:31:21 (3676): No heartbeat from core client for 30 sec - exiting 08:31:23 (3676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:24 (3676): No heartbeat from core client for 30 sec - exiting 08:31:25 (3676): No heartbeat from core client for 30 sec - exiting 08:31:27 (3676): No heartbeat from core client for 30 sec - exiting 11:24:58 (3568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:23:51 (10120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:22:43 (21500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:21:35 (34588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:57:11 (460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:19:59 (3188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:18:53 (12376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:08:43 (3088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:07:42 (15088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:59:53 (3388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:38:03 (1588): No heartbeat from core client for 30 sec - exiting 17:38:04 (1588): No heartbeat from core client for 30 sec - exiting 17:38:05 (1588): No heartbeat from core client for 30 sec - exiting 17:38:06 (1588): No heartbeat from core client for 30 sec - exiting 17:38:07 (1588): No heartbeat from core client for 30 sec - exiting 17:38:09 (1588): No heartbeat from core client for 30 sec - exiting 17:38:10 (1588): No heartbeat from core client for 30 sec - exiting 17:38:11 (1588): No heartbeat from core client for 30 sec - exiting 17:38:12 (1588): No heartbeat from core client for 30 sec - exiting 17:38:13 (1588): No heartbeat from core client for 30 sec - exiting 17:38:14 (1588): No heartbeat from core client for 30 sec - exiting 17:38:16 (1588): No heartbeat from core client for 30 sec - exiting 17:38:17 (1588): No heartbeat from core client for 30 sec - exiting 17:38:18 (1588): No heartbeat from core client for 30 sec - exiting 17:38:19 (1588): No heartbeat from core client for 30 sec - exiting 17:38:20 (1588): No heartbeat from core client for 30 sec - exiting 17:38:21 (1588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:38:22 (1588): No heartbeat from core client for 30 sec - exiting 17:38:24 (1588): No heartbeat from core client for 30 sec - exiting 17:38:25 (1588): No heartbeat from core client for 30 sec - exiting 17:38:26 (1588): No heartbeat from core client for 30 sec - exiting 20:32:04 (3764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:07:34 (3080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:59:39 (26456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:02:38 (3988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:01:16 (3220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:00:08 (25360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:59:05 (34188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:27:26 (2728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:26:13 (11164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:24:59 (22472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:23:55 (32408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:56:33 (756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:21 (10732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:54:11 (21128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:11:43 (3308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:24:54 (2976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:23:32 (11344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:22:26 (15436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:20:36 (3424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10448, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10448, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10448, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10448, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10448, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10448, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Dec 2012 20:02:36	270720	15465274	hadcm3n_o5ea_2100_40_008254521_0	155,520	718,327	4.6189
19 Dec 2012 15:33:07	270720	15465274	hadcm3n_o5ea_2100_40_008254521_0	129,600	602,118	4.6460
17 Dec 2012 17:41:26	270720	15465274	hadcm3n_o5ea_2100_40_008254521_0	103,680	482,649	4.6552
17 Dec 2012 17:41:26	270720	15465274	hadcm3n_o5ea_2100_40_008254521_0	77,760	363,911	4.6799
05 Dec 2012 14:55:47	270720	15465274	hadcm3n_o5ea_2100_40_008254521_0	51,840	243,163	4.6906
02 Dec 2012 14:39:42	270720	15465274	hadcm3n_o5ea_2100_40_008254521_0	25,920	122,433	4.7235