Task 16102677

Name	hadcm3n_8295_1980_40_008460860_1
Workunit	8611716
Created	3 Dec 2013, 8:34:52 UTC
Sent	3 Dec 2013, 8:34:59 UTC
Report deadline	4 Mar 2014, 16:02:10 UTC
Received	10 Dec 2013, 9:28:26 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1084861
Run time	5 days 16 hours 20 min 27 sec
CPU time	5 days 13 hours 27 min 22 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	3.04 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 05:15:00 (3512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:32 (7540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:33:29 (10108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:11:09 (10708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:05:16 (7200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:48:42 (10828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:49:41 (10188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:27:36 (5832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:30:37 (8548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:16:42 (7072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:17:20 (5268): No heartbeat from core client for 30 sec - exiting 16:17:21 (5268): No heartbeat from core client for 30 sec - exiting 16:17:22 (5268): No heartbeat from core client for 30 sec - exiting 16:17:23 (5268): No heartbeat from core client for 30 sec - exiting 16:17:24 (5268): No heartbeat from core client for 30 sec - exiting 16:17:25 (5268): No heartbeat from core client for 30 sec - exiting 16:17:26 (5268): No heartbeat from core client for 30 sec - exiting 16:17:28 (5268): No heartbeat from core client for 30 sec - exiting 16:17:29 (5268): No heartbeat from core client for 30 sec - exiting 16:17:30 (5268): No heartbeat from core client for 30 sec - exiting 16:17:31 (5268): No heartbeat from core client for 30 sec - exiting 16:17:32 (5268): No heartbeat from core client for 30 sec - exiting 16:17:33 (5268): No heartbeat from core client for 30 sec - exiting 16:17:34 (5268): No heartbeat from core client for 30 sec - exiting 16:17:35 (5268): No heartbeat from core client for 30 sec - exiting 16:17:36 (5268): No heartbeat from core client for 30 sec - exiting 16:17:37 (5268): No heartbeat from core client for 30 sec - exiting 16:17:38 (5268): No heartbeat from core client for 30 sec - exiting 16:17:40 (5268): No heartbeat from core client for 30 sec - exiting 16:17:41 (5268): No heartbeat from core client for 30 sec - exiting 16:17:42 (5268): No heartbeat from core client for 30 sec - exiting 16:17:43 (5268): No heartbeat from core client for 30 sec - exiting 16:17:44 (5268): No heartbeat from core client for 30 sec - exiting 16:17:45 (5268): No heartbeat from core client for 30 sec - exiting 16:17:46 (5268): No heartbeat from core client for 30 sec - exiting 16:17:47 (5268): No heartbeat from core client for 30 sec - exiting 16:17:48 (5268): No heartbeat from core client for 30 sec - exiting 16:17:49 (5268): No heartbeat from core client for 30 sec - exiting 16:17:50 (5268): No heartbeat from core client for 30 sec - exiting 16:17:52 (5268): No heartbeat from core client for 30 sec - exiting 16:17:53 (5268): No heartbeat from core client for 30 sec - exiting 16:17:54 (5268): No heartbeat from core client for 30 sec - exiting 16:17:55 (5268): No heartbeat from core client for 30 sec - exiting 16:17:56 (5268): No heartbeat from core client for 30 sec - exiting 16:17:57 (5268): No heartbeat from core client for 30 sec - exiting 16:17:58 (5268): No heartbeat from core client for 30 sec - exiting 16:17:59 (5268): No heartbeat from core client for 30 sec - exiting 16:18:00 (5268): No heartbeat from core client for 30 sec - exiting 16:18:01 (5268): No heartbeat from core client for 30 sec - exiting 16:18:02 (5268): No heartbeat from core client for 30 sec - exiting 16:18:04 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:12 (11060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:58 (9376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:06:03 (10784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:17:56 (6984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:07 (9864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:47:59 (7836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:48:41 (2512): No heartbeat from core client for 30 sec - exiting 18:48:42 (2512): No heartbeat from core client for 30 sec - exiting 18:48:43 (2512): No heartbeat from core client for 30 sec - exiting 18:48:44 (2512): No heartbeat from core client for 30 sec - exiting 18:48:45 (2512): No heartbeat from core client for 30 sec - exiting 18:48:46 (2512): No heartbeat from core client for 30 sec - exiting 18:48:47 (2512): No heartbeat from core client for 30 sec - exiting 18:48:48 (2512): No heartbeat from core client for 30 sec - exiting 18:48:50 (2512): No heartbeat from core client for 30 sec - exiting 18:48:51 (2512): No heartbeat from core client for 30 sec - exiting 18:48:52 (2512): No heartbeat from core client for 30 sec - exiting 18:48:53 (2512): No heartbeat from core client for 30 sec - exiting 18:48:54 (2512): No heartbeat from core client for 30 sec - exiting 18:48:55 (2512): No heartbeat from core client for 30 sec - exiting 18:48:56 (2512): No heartbeat from core client for 30 sec - exiting 18:48:57 (2512): No heartbeat from core client for 30 sec - exiting 18:48:58 (2512): No heartbeat from core client for 30 sec - exiting 18:48:59 (2512): No heartbeat from core client for 30 sec - exiting 18:49:00 (2512): No heartbeat from core client for 30 sec - exiting 18:49:02 (2512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:31:40 (4964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:12:37 (8892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:30 (9740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:47:19 (1988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:47:59 (11188): No heartbeat from core client for 30 sec - exiting 21:48:00 (11188): No heartbeat from core client for 30 sec - exiting 21:48:01 (11188): No heartbeat from core client for 30 sec - exiting 21:48:02 (11188): No heartbeat from core client for 30 sec - exiting 21:48:03 (11188): No heartbeat from core client for 30 sec - exiting 21:48:04 (11188): No heartbeat from core client for 30 sec - exiting 21:48:05 (11188): No heartbeat from core client for 30 sec - exiting 21:48:06 (11188): No heartbeat from core client for 30 sec - exiting 21:48:08 (11188): No heartbeat from core client for 30 sec - exiting 21:48:09 (11188): No heartbeat from core client for 30 sec - exiting 21:48:10 (11188): No heartbeat from core client for 30 sec - exiting 21:48:11 (11188): No heartbeat from core client for 30 sec - exiting 21:48:12 (11188): No heartbeat from core client for 30 sec - exiting 21:48:13 (11188): No heartbeat from core client for 30 sec - exiting 21:48:14 (11188): No heartbeat from core client for 30 sec - exiting 21:48:15 (11188): No heartbeat from core client for 30 sec - exiting 21:48:16 (11188): No heartbeat from core client for 30 sec - exiting 21:48:17 (11188): No heartbeat from core client for 30 sec - exiting 21:48:19 (11188): No heartbeat from core client for 30 sec - exiting 21:48:20 (11188): No heartbeat from core client for 30 sec - exiting 21:48:21 (11188): No heartbeat from core client for 30 sec - exiting 21:48:22 (11188): No heartbeat from core client for 30 sec - exiting 21:48:23 (11188): No heartbeat from core client for 30 sec - exiting 21:48:24 (11188): No heartbeat from core client for 30 sec - exiting 21:48:25 (11188): No heartbeat from core client for 30 sec - exiting 21:48:26 (11188): No heartbeat from core client for 30 sec - exiting 21:48:27 (11188): No heartbeat from core client for 30 sec - exiting 21:48:28 (11188): No heartbeat from core client for 30 sec - exiting 21:48:29 (11188): No heartbeat from core client for 30 sec - exiting 21:48:31 (11188): No heartbeat from core client for 30 sec - exiting 21:48:32 (11188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:31 (2980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:51:24 (5564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:23:40 (7228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:30:44 (11008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:09:00 (2012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:11 (3160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:50 (7504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:41:33 (8060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:14:51 (7380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:15:45 (4124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:28:36 (10500): No heartbeat from core client for 30 sec - exiting 21:28:37 (10500): No heartbeat from core client for 30 sec - exiting 21:28:38 (10500): No heartbeat from core client for 30 sec - exiting 21:28:39 (10500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:45:04 (8032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:47:30 (5732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:47 (8860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:59:07 (11092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:04:25 (8960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:35:23 (11132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:43:32 (8692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:58:29 (2620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:34:48 (10384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:00:28 (10644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:01:11 (7920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:32:15 (10748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:33:34 (9296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:10:22 (6032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:21:08 (11256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:21:54 (7004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:54:35 (11028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:36:59 (3300): No heartbeat from core client for 30 sec - exiting 10:37:00 (3300): No heartbeat from core client for 30 sec - exiting 10:37:01 (3300): No heartbeat from core client for 30 sec - exiting 10:37:02 (3300): No heartbeat from core client for 30 sec - exiting 10:37:03 (3300): No heartbeat from core client for 30 sec - exiting 10:37:04 (3300): No heartbeat from core client for 30 sec - exiting 10:37:05 (3300): No heartbeat from core client for 30 sec - exiting 10:37:06 (3300): No heartbeat from core client for 30 sec - exiting 10:37:07 (3300): No heartbeat from core client for 30 sec - exiting 10:37:08 (3300): No heartbeat from core client for 30 sec - exiting 10:37:09 (3300): No heartbeat from core client for 30 sec - exiting 10:37:11 (3300): No heartbeat from core client for 30 sec - exiting 10:37:12 (3300): No heartbeat from core client for 30 sec - exiting 10:37:13 (3300): No heartbeat from core client for 30 sec - exiting 10:37:14 (3300): No heartbeat from core client for 30 sec - exiting 10:37:15 (3300): No heartbeat from core client for 30 sec - exiting 10:37:16 (3300): No heartbeat from core client for 30 sec - exiting 10:37:17 (3300): No heartbeat from core client for 30 sec - exiting 10:37:18 (3300): No heartbeat from core client for 30 sec - exiting 10:37:19 (3300): No heartbeat from core client for 30 sec - exiting 10:37:20 (3300): No heartbeat from core client for 30 sec - exiting 10:37:22 (3300): No heartbeat from core client for 30 sec - exiting 10:37:23 (3300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3496, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3496, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3496, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3496, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3496, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3496, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Dec 2013 21:29:22	1084861	16102677	hadcm3n_8295_1980_40_008460860_1	155,520	418,041	2.6880
07 Dec 2013 19:03:01	1084861	16102677	hadcm3n_8295_1980_40_008460860_1	129,600	351,601	2.7130
06 Dec 2013 18:55:00	1084861	16102677	hadcm3n_8295_1980_40_008460860_1	103,680	282,162	2.7215
05 Dec 2013 22:33:20	1084861	16102677	hadcm3n_8295_1980_40_008460860_1	77,760	211,133	2.7152
05 Dec 2013 01:15:26	1084861	16102677	hadcm3n_8295_1980_40_008460860_1	51,840	139,798	2.6967
04 Dec 2013 04:43:48	1084861	16102677	hadcm3n_8295_1980_40_008460860_1	25,920	68,158	2.6296