Task 16586834

Name	hadcm3n_89qg_1980_40_008722035_0
Workunit	8868013
Created	23 Apr 2014, 12:42:51 UTC
Sent	4 May 2014, 19:06:47 UTC
Report deadline	4 Aug 2014, 2:33:58 UTC
Received	7 Jul 2014, 16:03:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1311491
Run time	8 days 12 hours 18 min 59 sec
CPU time	8 days 7 hours 39 min 52 sec
Validate state	Invalid
Credit	6,531.84
Device peak FLOPS	1.83 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:33:54 (2740): No heartbeat from core client for 30 sec - exiting 11:33:55 (2740): No heartbeat from core client for 30 sec - exiting 11:33:56 (2740): No heartbeat from core client for 30 sec - exiting 11:33:57 (2740): No heartbeat from core client for 30 sec - exiting 11:33:58 (2740): No heartbeat from core client for 30 sec - exiting 11:33:59 (2740): No heartbeat from core client for 30 sec - exiting 11:34:00 (2740): No heartbeat from core client for 30 sec - exiting 11:34:01 (2740): No heartbeat from core client for 30 sec - exiting 11:34:02 (2740): No heartbeat from core client for 30 sec - exiting 11:34:03 (2740): No heartbeat from core client for 30 sec - exiting 11:34:04 (2740): No heartbeat from core client for 30 sec - exiting 11:34:05 (2740): No heartbeat from core client for 30 sec - exiting 11:34:06 (2740): No heartbeat from core client for 30 sec - exiting 11:34:07 (2740): No heartbeat from core client for 30 sec - exiting 11:34:08 (2740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:11:33 (5448): No heartbeat from core client for 30 sec - exiting 10:11:34 (5448): No heartbeat from core client for 30 sec - exiting 10:11:35 (5448): No heartbeat from core client for 30 sec - exiting 10:11:36 (5448): No heartbeat from core client for 30 sec - exiting 10:11:37 (5448): No heartbeat from core client for 30 sec - exiting 10:11:38 (5448): No heartbeat from core client for 30 sec - exiting 10:11:39 (5448): No heartbeat from core client for 30 sec - exiting 10:11:40 (5448): No heartbeat from core client for 30 sec - exiting 10:11:41 (5448): No heartbeat from core client for 30 sec - exiting 10:11:42 (5448): No heartbeat from core client for 30 sec - exiting 10:11:43 (5448): No heartbeat from core client for 30 sec - exiting 10:11:44 (5448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:11:45 (5448): No heartbeat from core client for 30 sec - exiting 10:11:46 (5448): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1828, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1828, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1828, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1828, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1828, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1828, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jul 2014 02:23:47	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	544,320	701,665	1.2891
06 Jul 2014 14:26:26	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	518,400	668,578	1.2897
06 Jul 2014 04:46:33	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	492,480	635,326	1.2901
05 Jul 2014 19:03:56	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	466,560	600,731	1.2876
05 Jul 2014 05:09:31	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	440,640	566,352	1.2853
04 Jul 2014 19:56:51	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	414,720	532,713	1.2845
03 Jul 2014 16:37:38	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	388,800	504,684	1.2981
23 Jun 2014 12:04:35	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	362,880	470,697	1.2971
23 Jun 2014 12:04:35	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	336,960	437,404	1.2981
23 Jun 2014 12:04:35	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	311,040	404,069	1.2991
21 Jun 2014 15:46:37	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	285,120	370,676	1.3001
21 Jun 2014 06:30:13	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	259,200	337,290	1.3013
20 Jun 2014 19:48:29	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	233,280	303,392	1.3005
20 Jun 2014 09:16:30	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	207,360	269,408	1.2992
20 Jun 2014 00:24:50	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	181,440	236,116	1.3013
19 Jun 2014 14:07:34	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	155,520	202,313	1.3009
19 Jun 2014 01:45:05	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	129,600	168,549	1.3005
18 Jun 2014 16:13:12	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	103,680	135,077	1.3028
18 Jun 2014 06:38:57	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	77,760	101,981	1.3115
03 Jun 2014 19:48:40	1311491	16586834	hadcm3n_89qg_1980_40_008722035_0	51,840	67,833	1.3085