Task 15881478

Name	hadcm3n_4fsk_1940_40_008307148_3
Workunit	8458283
Created	5 Jul 2013, 3:19:49 UTC
Sent	5 Jul 2013, 3:50:49 UTC
Report deadline	4 Oct 2013, 11:18:00 UTC
Received	17 Sep 2013, 11:21:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1237173
Run time	2 days 22 hours 36 min 41 sec
CPU time	2 days 8 hours 35 min 50 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	3.37 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 13:35:38 (31024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:11:45 (32800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:39:11 (33828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:39:14 (33828): No heartbeat from core client for 30 sec - exiting 19:39:15 (33828): No heartbeat from core client for 30 sec - exiting 19:39:16 (33828): No heartbeat from core client for 30 sec - exiting 19:39:17 (33828): No heartbeat from core client for 30 sec - exiting 19:39:18 (33828): No heartbeat from core client for 30 sec - exiting 23:03:21 (32736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:03:30 (32736): No heartbeat from core client for 30 sec - exiting 23:03:31 (32736): No heartbeat from core client for 30 sec - exiting 23:03:32 (32736): No heartbeat from core client for 30 sec - exiting 23:03:33 (32736): No heartbeat from core client for 30 sec - exiting 23:03:34 (32736): No heartbeat from core client for 30 sec - exiting 23:03:35 (32736): No heartbeat from core client for 30 sec - exiting 23:03:36 (32736): No heartbeat from core client for 30 sec - exiting 03:57:28 (36804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:38:21 (34588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:38:22 (34588): No heartbeat from core client for 30 sec - exiting 04:38:23 (34588): No heartbeat from core client for 30 sec - exiting 09:38:42 (2676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:38:51 (2676): No heartbeat from core client for 30 sec - exiting 09:38:52 (2676): No heartbeat from core client for 30 sec - exiting 10:06:57 (18548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 10:49:36 (25376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:49:37 (25376): No heartbeat from core client for 30 sec - exiting 22:23:24 (7032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:49:05 (12380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:19 (35192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:23 (35192): No heartbeat from core client for 30 sec - exiting 16:55:24 (35192): No heartbeat from core client for 30 sec - exiting 16:55:25 (35192): No heartbeat from core client for 30 sec - exiting 16:55:26 (35192): No heartbeat from core client for 30 sec - exiting 16:55:27 (35192): No heartbeat from core client for 30 sec - exiting 16:55:28 (35192): No heartbeat from core client for 30 sec - exiting 16:55:29 (35192): No heartbeat from core client for 30 sec - exiting 16:55:30 (35192): No heartbeat from core client for 30 sec - exiting 16:55:31 (35192): No heartbeat from core client for 30 sec - exiting 16:55:32 (35192): No heartbeat from core client for 30 sec - exiting 16:58:17 (29564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:05:38 (36960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:05:49 (36960): No heartbeat from core client for 30 sec - exiting 17:05:50 (36960): No heartbeat from core client for 30 sec - exiting 17:05:51 (36960): No heartbeat from core client for 30 sec - exiting 17:05:52 (36960): No heartbeat from core client for 30 sec - exiting 17:05:53 (36960): No heartbeat from core client for 30 sec - exiting 17:05:54 (36960): No heartbeat from core client for 30 sec - exiting 17:05:55 (36960): No heartbeat from core client for 30 sec - exiting 17:05:56 (36960): No heartbeat from core client for 30 sec - exiting 17:05:57 (36960): No heartbeat from core client for 30 sec - exiting 17:05:58 (36960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 07:01:39 (4336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:02:01 (4336): No heartbeat from core client for 30 sec - exiting 07:02:02 (4336): No heartbeat from core client for 30 sec - exiting 07:02:03 (4336): No heartbeat from core client for 30 sec - exiting 07:02:04 (4336): No heartbeat from core client for 30 sec - exiting 07:02:05 (4336): No heartbeat from core client for 30 sec - exiting 07:02:06 (4336): No heartbeat from core client for 30 sec - exiting 07:02:07 (4336): No heartbeat from core client for 30 sec - exiting 07:02:08 (4336): No heartbeat from core client for 30 sec - exiting 07:02:09 (4336): No heartbeat from core client for 30 sec - exiting 07:02:10 (4336): No heartbeat from core client for 30 sec - exiting 07:02:11 (4336): No heartbeat from core client for 30 sec - exiting 07:02:12 (4336): No heartbeat from core client for 30 sec - exiting 07:02:13 (4336): No heartbeat from core client for 30 sec - exiting 07:03:50 (6516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:03 (13424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:07:44 (11276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:08 (13416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:10:31 (14220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:55:39 (6588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:57:25 (28972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:01:03 (27584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:03:56 (29856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:49:34 (288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:55:16 (32356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:02 (33036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:59:30 (34664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=35688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=35688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=35688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=35688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=35688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=35688, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish CPDN Monitor - Quit request from BOINC... 22:41:37 (30444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:00:42 (80152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:00:50 (80152): No heartbeat from core client for 30 sec - exiting 23:00:51 (80152): No heartbeat from core client for 30 sec - exiting 23:07:15 (18124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:55:03 (51556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:09:39 (96112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:41:45 (90304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:49:35 (21336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:36 (57292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:12 (99240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:40:30 (93732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:59:20 (89756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:40:54 (44420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:40:58 (44420): No heartbeat from core client for 30 sec - exiting 04:40:59 (44420): No heartbeat from core client for 30 sec - exiting 04:41:00 (44420): No heartbeat from core client for 30 sec - exiting 04:41:01 (44420): No heartbeat from core client for 30 sec - exiting 04:41:02 (44420): No heartbeat from core client for 30 sec - exiting 04:41:03 (44420): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 06:24:57 (100048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:58 (100048): No heartbeat from core client for 30 sec - exiting 06:31:17 (54900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:31:18 (54900): No heartbeat from core client for 30 sec - exiting 08:26:44 (55476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:38:37 (108448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:38:46 (108448): No heartbeat from core client for 30 sec - exiting 10:38:47 (108448): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 12:25:19 (35784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:17:45 (80104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:09:15 (86040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:54:52 (37332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:34:10 (24024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:34:13 (24024): No heartbeat from core client for 30 sec - exiting 15:34:14 (24024): No heartbeat from core client for 30 sec - exiting 15:34:15 (24024): No heartbeat from core client for 30 sec - exiting 15:34:16 (24024): No heartbeat from core client for 30 sec - exiting 15:34:17 (24024): No heartbeat from core client for 30 sec - exiting 15:34:18 (24024): No heartbeat from core client for 30 sec - exiting 15:34:19 (24024): No heartbeat from core client for 30 sec - exiting 15:34:20 (24024): No heartbeat from core client for 30 sec - exiting 15:34:21 (24024): No heartbeat from core client for 30 sec - exiting 17:40:26 (45964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:12 (30188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:06:48 (27056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:36:33 (88044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:53:44 (47512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:57:02 (48956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:34:33 (39280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:35:12 (86560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:47:40 (89604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:20:17 (89128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:35:07 (88936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:28:07 (34760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:55:54 (24568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:55:33 (37360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:35:43 (103236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:55:47 (76708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:18 (94752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:15:27 (39028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:55:36 (64476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:03:15 (56384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:59:13 (51420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:15:22 (17968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:57:31 (107704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:42:40 (31700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:43:14 (86988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:28:29 (45664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:44:14 (93564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:17 (92532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:04 (77756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:25:00 (106508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:45:20 (107960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:08:11 (42696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:18:28 (35740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:09:11 (33192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=79804, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=79804, iMonCtr=1 Model crash detected, will try to restart... 18:13:21 (79804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:53:10 (69348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:32:06 (69712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:26 (72044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:43:35 (72896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:01:31 (72712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:13:30 (72892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:26:15 (75440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:49:25 (68776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=76336, iMonCtr=1 Model crash detected, will try to restart... 17:20:30 (76336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:34:29 (73100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:36:39 (74944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:39:04 (79696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:46:57 (79380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:49:07 (77432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:41:11 (79876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:41:17 (79876): No heartbeat from core client for 30 sec - exiting 00:35:15 (75124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:10:33 (78840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:45:33 (79964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=81520, iMonCtr=1 Model crash detected, will try to restart... 02:47:25 (81520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... S02:51:38 (76272): No heartbeat from core client for 30 sec - exiting ignal 22 received, exiting... Called boinc_finish CPDN Monitor - No 'heartbeat' from BOINC... Si2:56:45 (82544): No heartbeat from core client for 30 sec - exiting gnal 22 received, exiting... Called boinc_finish CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=79284, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=79284, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Aug 2013 15:39:48	1237173	15881478	hadcm3n_4fsk_1940_40_008307148_3	103,680	118,232	1.1404
14 Aug 2013 15:39:48	1237173	15881478	hadcm3n_4fsk_1940_40_008307148_3	77,760	89,776	1.1545
14 Aug 2013 15:39:48	1237173	15881478	hadcm3n_4fsk_1940_40_008307148_3	51,840	60,334	1.1639
14 Aug 2013 15:39:48	1237173	15881478	hadcm3n_4fsk_1940_40_008307148_3	25,920	29,928	1.1546