Task 13090914

Name	hadcm3n_y88a_1900_40_007343524_1
Workunit	7540954
Created	6 Jul 2011, 13:18:41 UTC
Sent	23 Jul 2011, 1:51:52 UTC
Report deadline	22 Oct 2011, 9:19:03 UTC
Received	17 Dec 2011, 6:06:15 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1141343
Run time	12 days 16 hours 12 min 33 sec
CPU time	11 days 11 hours 4 min 46 sec
Validate state	Invalid
Credit	5,598.72
Device peak FLOPS	2.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... 17:00:55 (4068): No heartbeat from core client for 30 sec - exiting 17:00:56 (4068): No heartbeat from core client for 30 sec - exiting 17:00:57 (4068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:58 (4068): No heartbeat from core client for 30 sec - exiting 17:00:59 (4068): No heartbeat from core client for 30 sec - exiting 17:01:00 (4068): No heartbeat from core client for 30 sec - exiting 20:01:59 (1100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:21:48 (3476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:42:35 (3456): No heartbeat from core client for 30 sec - exiting 10:42:36 (3456): No heartbeat from core client for 30 sec - exiting 10:42:37 (3456): No heartbeat from core client for 30 sec - exiting 10:42:38 (3456): No heartbeat from core client for 30 sec - exiting 10:42:39 (3456): No heartbeat from core client for 30 sec - exiting 10:42:40 (3456): No heartbeat from core client for 30 sec - exiting 10:42:41 (3456): No heartbeat from core client for 30 sec - exiting 10:42:42 (3456): No heartbeat from core client for 30 sec - exiting 10:42:43 (3456): No heartbeat from core client for 30 sec - exiting 10:42:44 (3456): No heartbeat from core client for 30 sec - exiting 10:42:45 (3456): No heartbeat from core client for 30 sec - exiting 10:42:46 (3456): No heartbeat from core client for 30 sec - exiting 10:42:47 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2664, iMonCtr=1 Model crash detected, will try to restart... 10:54:49 (4124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:40:04 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:32:44 (3808): No heartbeat from core client for 30 sec - exiting 17:32:46 (3808): No heartbeat from core client for 30 sec - exiting 17:32:47 (3808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:59:20 (3852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:05:11 (4076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:41:53 (4052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:33:04 (3652): No heartbeat from core client for 30 sec - exiting 19:33:05 (3652): No heartbeat from core client for 30 sec - exiting 19:33:06 (3652): No heartbeat from core client for 30 sec - exiting 19:33:07 (3652): No heartbeat from core client for 30 sec - exiting 19:33:08 (3652): No heartbeat from core client for 30 sec - exiting 19:33:09 (3652): No heartbeat from core client for 30 sec - exiting 19:33:10 (3652): No heartbeat from core client for 30 sec - exiting 19:33:11 (3652): No heartbeat from core client for 30 sec - exiting 19:33:12 (3652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:13 (3652): No heartbeat from core client for 30 sec - exiting 19:33:14 (3652): No heartbeat from core client for 30 sec - exiting 19:33:15 (3652): No heartbeat from core client for 30 sec - exiting 19:33:16 (3652): No heartbeat from core client for 30 sec - exiting 19:33:17 (3652): No heartbeat from core client for 30 sec - exiting 19:33:18 (3652): No heartbeat from core client for 30 sec - exiting 09:44:10 (4176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:08:51 (3780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:01:33 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3820, iMonCtr=1 Model crash detected, will try to restart... 14:02:19 (4652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:16:54 (3756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:19:11 (3396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:18:20 (4316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:45:58 (2024): No heartbeat from core client for 30 sec - exiting 09:45:59 (2024): No heartbeat from core client for 30 sec - exiting 09:46:00 (2024): No heartbeat from core client for 30 sec - exiting 09:46:01 (2024): No heartbeat from core client for 30 sec - exiting 09:46:02 (2024): No heartbeat from core client for 30 sec - exiting 09:46:03 (2024): No heartbeat from core client for 30 sec - exiting 09:46:04 (2024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:06:40 (4364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:34:23 (3964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:49:33 (432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:39:14 (2040): No heartbeat from core client for 30 sec - exiting 10:39:15 (2040): No heartbeat from core client for 30 sec - exiting 10:39:16 (2040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:41:03 (4612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=336, iMonCtr=1 Model crash detected, will try to restart... 19:55:21 (3992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:19:01 (3660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:21:20 (4764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3620, iMonCtr=1 Model crash detected, will try to restart... 10:18:17 (4228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:15:57 (4092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:51:18 (3424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:14:01 (3740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:10:15 (4284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:21:01 (3712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:39:23 (4772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:50:24 (3852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:53:57 (1560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4796, iMonCtr=1 Model crash detected, will try to restart... 14:53:01 (4188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:21:25 (3036): No heartbeat from core client for 30 sec - exiting 19:21:26 (3036): No heartbeat from core client for 30 sec - exiting 19:21:27 (3036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:13:39 (4000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:40:44 (3812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:32:33 (3624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:13:23 (4300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:56:21 (4308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:42:27 (3692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:13:47 (176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... 11:42:23 (2852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:32:06 (2948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:46:36 (4564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:21:06 (4652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5116, iMonCtr=1 Model crash detected, will try to restart... 18:52:51 (4040): No heartbeat from core client for 30 sec - exiting 18:52:52 (4040): No heartbeat from core client for 30 sec - exiting 18:52:53 (4040): No heartbeat from core client for 30 sec - exiting 18:52:54 (4040): No heartbeat from core client for 30 sec - exiting 18:52:55 (4040): No heartbeat from core client for 30 sec - exiting 18:52:56 (4040): No heartbeat from core client for 30 sec - exiting 18:52:57 (4040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:44:06 (2804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:50:54 (196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:04:15 (3720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:29:03 (4076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:01:40 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:15:51 (4504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:08:16 (3844): No heartbeat from core client for 30 sec - exiting 16:08:17 (3844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:25:01 (2740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:21:12 (1716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:09:19 (4932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:54:03 (3880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:24:19 (3740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:09:52 (4056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4804, iMonCtr=1 Model crash detected, will try to restart... 09:21:45 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:14:04 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:27:30 (3412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:47:20 (2640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:49 (2536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:50:58 (4184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:52:32 (5032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... C15:34:52 (4084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:34:53 (4084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:58:47 (3404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5116, iMonCtr=1 Model crash detected, will try to restart... 20:02:51 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:30:23 (4252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:34:19 (4248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4724, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4724, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4724, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4372, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4372, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 22:03:17 (792): No heartbeat from core client for 30 sec - exiting 22:03:18 (792): No heartbeat from core client for 30 sec - exiting 22:03:19 (792): No heartbeat from core client for 30 sec - exiting 22:03:20 (792): No heartbeat from core client for 30 sec - exiting 22:03:21 (792): No heartbeat from core client for 30 sec - exiting 22:03:22 (792): No heartbeat from core client for 30 sec - exiting 22:03:23 (792): No heartbeat from core client for 30 sec - exiting 22:03:24 (792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4956, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Dec 2011 17:04:15	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	466,560	966,879	2.0724
02 Dec 2011 02:25:14	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	440,640	916,385	2.0797
23 Nov 2011 09:07:57	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	414,720	864,667	2.0849
16 Nov 2011 18:36:43	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	388,800	810,028	2.0834
02 Nov 2011 21:02:42	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	362,880	757,071	2.0863
13 Oct 2011 03:33:59	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	336,960	702,564	2.0850
05 Oct 2011 21:57:59	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	311,040	648,002	2.0833
26 Sep 2011 06:18:15	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	285,120	593,674	2.0822
14 Sep 2011 02:30:10	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	259,200	540,886	2.0868
11 Sep 2011 16:02:09	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	233,280	487,782	2.0910
09 Sep 2011 00:23:30	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	207,360	435,108	2.0983
28 Aug 2011 14:03:31	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	181,440	381,928	2.1050
27 Aug 2011 07:21:17	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	155,520	330,139	2.1228
17 Aug 2011 00:58:24	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	129,600	276,155	2.1308
10 Aug 2011 03:56:41	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	103,680	216,567	2.0888
06 Aug 2011 07:01:46	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	77,760	162,962	2.0957
04 Aug 2011 23:44:55	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	51,840	107,869	2.0808
28 Jul 2011 21:41:27	1141343	13090914	hadcm3n_y88a_1900_40_007343524_1	25,920	53,734	2.0731