Task 17909126

Name	hadcm3n_xa7c_1940_40_009149518_1
Workunit	9279854
Created	7 Feb 2015, 23:07:28 UTC
Sent	7 Feb 2015, 23:08:07 UTC
Report deadline	10 May 2015, 6:35:18 UTC
Received	26 Feb 2015, 11:34:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1068742
Run time	14 days 1 hours 22 min 36 sec
CPU time	13 days 3 hours 1 min 55 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	2.90 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.4.27</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:42:43 (6588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:00:49 (6312): No heartbeat from core client for 30 sec - exiting 19:00:50 (6312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:55:07 (196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:26:04 (7352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17088, iMonCtr=1 Model crash detected, will try to restart... 03:24:32 (6704): No heartbeat from core client for 30 sec - exiting 03:24:33 (6704): No heartbeat from core client for 30 sec - exiting 03:24:34 (6704): No heartbeat from core client for 30 sec - exiting 03:24:35 (6704): No heartbeat from core client for 30 sec - exiting 03:24:36 (6704): No heartbeat from core client for 30 sec - exiting 03:24:37 (6704): No heartbeat from core client for 30 sec - exiting 03:24:38 (6704): No heartbeat from core client for 30 sec - exiting 03:24:39 (6704): No heartbeat from core client for 30 sec - exiting 03:24:40 (6704): No heartbeat from core client for 30 sec - exiting 03:24:41 (6704): No heartbeat from core client for 30 sec - exiting 03:24:42 (6704): No heartbeat from core client for 30 sec - exiting 03:24:43 (6704): No heartbeat from core client for 30 sec - exiting 03:24:44 (6704): No heartbeat from core client for 30 sec - exiting 03:24:45 (6704): No heartbeat from core client for 30 sec - exiting 03:24:46 (6704): No heartbeat from core client for 30 sec - exiting 03:24:47 (6704): No heartbeat from core client for 30 sec - exiting 03:24:48 (6704): No heartbeat from core client for 30 sec - exiting 03:24:49 (6704): No heartbeat from core client for 30 sec - exiting 03:24:50 (6704): No heartbeat from core client for 30 sec - exiting 03:24:51 (6704): No heartbeat from core client for 30 sec - exiting 03:24:52 (6704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:24:53 (6704): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:49:42 (5652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:55:54 (6348): No heartbeat from core client for 30 sec - exiting 19:55:55 (6348): No heartbeat from core client for 30 sec - exiting 19:55:56 (6348): No heartbeat from core client for 30 sec - exiting 19:55:57 (6348): No heartbeat from core client for 30 sec - exiting 19:55:58 (6348): No heartbeat from core client for 30 sec - exiting 19:55:59 (6348): No heartbeat from core client for 30 sec - exiting 19:56:00 (6348): No heartbeat from core client for 30 sec - exiting 19:56:01 (6348): No heartbeat from core client for 30 sec - exiting 19:56:02 (6348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:01:36 (6908): No heartbeat from core client for 30 sec - exiting 21:01:37 (6908): No heartbeat from core client for 30 sec - exiting 21:01:38 (6908): No heartbeat from core client for 30 sec - exiting 21:01:39 (6908): No heartbeat from core client for 30 sec - exiting 21:01:40 (6908): No heartbeat from core client for 30 sec - exiting 21:01:41 (6908): No heartbeat from core client for 30 sec - exiting 21:01:42 (6908): No heartbeat from core client for 30 sec - exiting 21:01:43 (6908): No heartbeat from core client for 30 sec - exiting 21:01:44 (6908): No heartbeat from core client for 30 sec - exiting 21:01:45 (6908): No heartbeat from core client for 30 sec - exiting 21:01:46 (6908): No heartbeat from core client for 30 sec - exiting 21:01:47 (6908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:45:36 (9764): No heartbeat from core client for 30 sec - exiting 20:45:38 (9764): No heartbeat from core client for 30 sec - exiting 20:45:39 (9764): No heartbeat from core client for 30 sec - exiting 20:45:40 (9764): No heartbeat from core client for 30 sec - exiting 20:45:41 (9764): No heartbeat from core client for 30 sec - exiting 20:45:42 (9764): No heartbeat from core client for 30 sec - exiting 20:45:43 (9764): No heartbeat from core client for 30 sec - exiting 20:45:44 (9764): No heartbeat from core client for 30 sec - exiting 20:45:45 (9764): No heartbeat from core client for 30 sec - exiting 20:45:46 (9764): No heartbeat from core client for 30 sec - exiting 20:45:47 (9764): No heartbeat from core client for 30 sec - exiting 20:45:48 (9764): No heartbeat from core client for 30 sec - exiting 20:45:49 (9764): No heartbeat from core client for 30 sec - exiting 20:45:50 (9764): No heartbeat from core client for 30 sec - exiting 20:45:51 (9764): No heartbeat from core client for 30 sec - exiting 20:45:52 (9764): No heartbeat from core client for 30 sec - exiting 20:45:53 (9764): No heartbeat from core client for 30 sec - exiting 20:45:54 (9764): No heartbeat from core client for 30 sec - exiting 20:45:55 (9764): No heartbeat from core client for 30 sec - exiting 20:45:56 (9764): No heartbeat from core client for 30 sec - exiting 20:45:57 (9764): No heartbeat from core client for 30 sec - exiting 20:45:58 (9764): No heartbeat from core client for 30 sec - exiting 20:45:59 (9764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9728, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9728, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9728, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9728, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9728, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9728, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9728, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9728, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9728, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Feb 2015 23:45:26	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	596,160	1,136,479	1.9063
24 Feb 2015 16:26:10	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	570,240	1,087,201	1.9066
24 Feb 2015 02:35:39	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	544,320	1,037,258	1.9056
23 Feb 2015 09:56:00	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	518,400	987,882	1.9056
22 Feb 2015 19:58:12	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	492,480	938,602	1.9059
21 Feb 2015 23:40:16	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	466,560	888,953	1.9053
20 Feb 2015 23:56:16	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	440,640	839,103	1.9043
20 Feb 2015 08:07:03	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	414,720	789,314	1.9032
19 Feb 2015 14:44:34	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	388,800	739,661	1.9024
18 Feb 2015 23:10:45	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	362,880	689,970	1.9014
18 Feb 2015 06:37:22	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	336,960	640,614	1.9012
17 Feb 2015 09:12:45	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	311,040	591,334	1.9012
16 Feb 2015 14:52:05	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	285,120	541,871	1.9005
16 Feb 2015 00:06:49	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	259,200	492,270	1.8992
15 Feb 2015 03:14:06	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	233,280	442,304	1.8960
14 Feb 2015 06:27:28	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	207,360	392,688	1.8938
13 Feb 2015 07:08:59	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	181,440	343,446	1.8929
12 Feb 2015 09:07:02	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	155,520	293,839	1.8894
11 Feb 2015 16:23:43	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	129,600	244,911	1.8897
11 Feb 2015 01:21:58	1068742	17909126	hadcm3n_xa7c_1940_40_009149518_1	103,680	196,748	1.8976