Task 16135498

Name	hadcm3n_810g_1980_40_008459251_2
Workunit	8610107
Created	6 Dec 2013, 22:04:52 UTC
Sent	6 Dec 2013, 22:33:36 UTC
Report deadline	8 Mar 2014, 6:00:47 UTC
Received	5 Jan 2014, 17:18:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1260309
Run time	7 days 0 hours 2 min 54 sec
CPU time	6 days 22 hours 0 min 24 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:58:40 (8756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:16:15 (9560): No heartbeat from core client for 30 sec - exiting 14:16:16 (9560): No heartbeat from core client for 30 sec - exiting 14:16:17 (9560): No heartbeat from core client for 30 sec - exiting 14:16:18 (9560): No heartbeat from core client for 30 sec - exiting 14:16:19 (9560): No heartbeat from core client for 30 sec - exiting 14:16:20 (9560): No heartbeat from core client for 30 sec - exiting 14:16:21 (9560): No heartbeat from core client for 30 sec - exiting 14:16:22 (9560): No heartbeat from core client for 30 sec - exiting 14:16:23 (9560): No heartbeat from core client for 30 sec - exiting 14:16:24 (9560): No heartbeat from core client for 30 sec - exiting 14:16:25 (9560): No heartbeat from core client for 30 sec - exiting 14:16:26 (9560): No heartbeat from core client for 30 sec - exiting 14:16:27 (9560): No heartbeat from core client for 30 sec - exiting 14:16:28 (9560): No heartbeat from core client for 30 sec - exiting 14:16:29 (9560): No heartbeat from core client for 30 sec - exiting 14:16:30 (9560): No heartbeat from core client for 30 sec - exiting 14:16:31 (9560): No heartbeat from core client for 30 sec - exiting 14:16:32 (9560): No heartbeat from core client for 30 sec - exiting 14:16:33 (9560): No heartbeat from core client for 30 sec - exiting 14:16:34 (9560): No heartbeat from core client for 30 sec - exiting 14:16:35 (9560): No heartbeat from core client for 30 sec - exiting 14:16:36 (9560): No heartbeat from core client for 30 sec - exiting 14:16:37 (9560): No heartbeat from core client for 30 sec - exiting 14:16:38 (9560): No heartbeat from core client for 30 sec - exiting 14:16:39 (9560): No heartbeat from core client for 30 sec - exiting 14:16:40 (9560): No heartbeat from core client for 30 sec - exiting 14:16:41 (9560): No heartbeat from core client for 30 sec - exiting 14:16:42 (9560): No heartbeat from core client for 30 sec - exiting 14:16:43 (9560): No heartbeat from core client for 30 sec - exiting 14:16:44 (9560): No heartbeat from core client for 30 sec - exiting 14:16:45 (9560): No heartbeat from core client for 30 sec - exiting 14:16:46 (9560): No heartbeat from core client for 30 sec - exiting 14:16:47 (9560): No heartbeat from core client for 30 sec - exiting 14:16:48 (9560): No heartbeat from core client for 30 sec - exiting 14:16:49 (9560): No heartbeat from core client for 30 sec - exiting 14:16:50 (9560): No heartbeat from core client for 30 sec - exiting 14:16:51 (9560): No heartbeat from core client for 30 sec - exiting 14:16:52 (9560): No heartbeat from core client for 30 sec - exiting 14:16:53 (9560): No heartbeat from core client for 30 sec - exiting 14:16:54 (9560): No heartbeat from core client for 30 sec - exiting 14:16:55 (9560): No heartbeat from core client for 30 sec - exiting 14:16:56 (9560): No heartbeat from core client for 30 sec - exiting 14:16:57 (9560): No heartbeat from core client for 30 sec - exiting 14:16:58 (9560): No heartbeat from core client for 30 sec - exiting 14:16:59 (9560): No heartbeat from core client for 30 sec - exiting 14:17:00 (9560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:34:06 (8956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:59:21 (8552): No heartbeat from core client for 30 sec - exiting 17:59:22 (8552): No heartbeat from core client for 30 sec - exiting 17:59:23 (8552): No heartbeat from core client for 30 sec - exiting 17:59:24 (8552): No heartbeat from core client for 30 sec - exiting 17:59:25 (8552): No heartbeat from core client for 30 sec - exiting 17:59:26 (8552): No heartbeat from core client for 30 sec - exiting 17:59:27 (8552): No heartbeat from core client for 30 sec - exiting 17:59:28 (8552): No heartbeat from core client for 30 sec - exiting 17:59:29 (8552): No heartbeat from core client for 30 sec - exiting 17:59:30 (8552): No heartbeat from core client for 30 sec - exiting 17:59:31 (8552): No heartbeat from core client for 30 sec - exiting 17:59:32 (8552): No heartbeat from core client for 30 sec - exiting 17:59:33 (8552): No heartbeat from core client for 30 sec - exiting 17:59:34 (8552): No heartbeat from core client for 30 sec - exiting 17:59:35 (8552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:45 (12272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5180, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:16:03 (9108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:15:22 (9680): No heartbeat from core client for 30 sec - exiting 19:15:23 (9680): No heartbeat from core client for 30 sec - exiting 19:15:24 (9680): No heartbeat from core client for 30 sec - exiting 19:15:25 (9680): No heartbeat from core client for 30 sec - exiting 19:15:26 (9680): No heartbeat from core client for 30 sec - exiting 19:15:27 (9680): No heartbeat from core client for 30 sec - exiting 19:15:28 (9680): No heartbeat from core client for 30 sec - exiting 19:15:29 (9680): No heartbeat from core client for 30 sec - exiting 19:15:30 (9680): No heartbeat from core client for 30 sec - exiting 19:15:31 (9680): No heartbeat from core client for 30 sec - exiting 19:15:32 (9680): No heartbeat from core client for 30 sec - exiting 19:15:33 (9680): No heartbeat from core client for 30 sec - exiting 19:15:34 (9680): No heartbeat from core client for 30 sec - exiting 19:15:35 (9680): No heartbeat from core client for 30 sec - exiting 19:15:36 (9680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5232, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9280, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9844, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10092, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10092, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 12:16:07 (8384): No heartbeat from core client for 30 sec - exiting 12:16:08 (8384): No heartbeat from core client for 30 sec - exiting 12:16:09 (8384): No heartbeat from core client for 30 sec - exiting 12:16:10 (8384): No heartbeat from core client for 30 sec - exiting 12:16:11 (8384): No heartbeat from core client for 30 sec - exiting 12:16:13 (8384): No heartbeat from core client for 30 sec - exiting 12:16:14 (8384): No heartbeat from core client for 30 sec - exiting 12:16:15 (8384): No heartbeat from core client for 30 sec - exiting 12:16:16 (8384): No heartbeat from core client for 30 sec - exiting 12:16:17 (8384): No heartbeat from core client for 30 sec - exiting 12:16:18 (8384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3092, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3092, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3092, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Dec 2013 07:22:04	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	440,640	562,228	1.2759
28 Dec 2013 21:50:16	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	414,720	528,162	1.2735
28 Dec 2013 12:26:25	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	388,800	494,241	1.2712
28 Dec 2013 02:49:07	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	362,880	459,865	1.2673
27 Dec 2013 12:17:03	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	336,960	426,229	1.2649
27 Dec 2013 02:55:24	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	311,040	392,785	1.2628
26 Dec 2013 17:32:21	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	285,120	358,937	1.2589
25 Dec 2013 15:52:26	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	259,200	325,303	1.2550
24 Dec 2013 17:41:08	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	233,280	292,241	1.2527
23 Dec 2013 17:08:33	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	207,360	260,385	1.2557
20 Dec 2013 04:14:31	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	181,440	227,680	1.2549
19 Dec 2013 06:04:37	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	155,520	195,016	1.2540
17 Dec 2013 20:02:39	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	129,600	161,902	1.2492
17 Dec 2013 03:09:22	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	103,680	130,008	1.2539
16 Dec 2013 17:05:11	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	77,760	98,366	1.2650
12 Dec 2013 19:29:16	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	51,840	65,219	1.2581
11 Dec 2013 21:19:03	1260309	16135498	hadcm3n_810g_1980_40_008459251_2	25,920	32,355	1.2483