Task 16294756

Name	hadcm3n_866a_1980_40_008514505_0
Workunit	8662017
Created	26 Feb 2014, 16:02:27 UTC
Sent	26 Feb 2014, 16:03:38 UTC
Report deadline	28 May 2014, 23:30:49 UTC
Received	23 Jun 2014, 13:43:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1294522
Run time	12 days 9 hours 2 min 49 sec
CPU time	11 days 7 hours 50 min 25 sec
Validate state	Invalid
Credit	7,776.00
Device peak FLOPS	2.83 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4712, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5996, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5460, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5092, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4220, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5868, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5048, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4652, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5888, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5868, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3632, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5988, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5060, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5328, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1252, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4436, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4572, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:59:32 (5908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4532, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4368, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4368, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5264, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5056, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Jun 2014 17:38:09	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	648,000	959,419	1.4806
19 Jun 2014 15:12:48	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	622,080	920,829	1.4802
17 Jun 2014 21:02:34	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	596,160	882,449	1.4802
16 Jun 2014 18:44:21	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	570,240	844,558	1.4811
10 Jun 2014 21:20:52	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	544,320	806,524	1.4817
04 Apr 2014 21:32:49	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	518,400	769,300	1.4840
03 Apr 2014 18:36:51	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	492,480	730,607	1.4835
02 Apr 2014 16:24:52	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	466,560	691,918	1.4830
01 Apr 2014 13:42:41	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	440,640	653,086	1.4821
28 Mar 2014 19:48:54	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	414,720	614,548	1.4818
27 Mar 2014 17:10:33	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	388,800	575,573	1.4804
26 Mar 2014 13:40:53	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	362,880	536,806	1.4793
24 Mar 2014 20:19:27	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	336,960	498,076	1.4781
21 Mar 2014 17:32:06	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	311,040	459,368	1.4769
19 Mar 2014 20:18:30	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	285,120	421,187	1.4772
18 Mar 2014 17:24:34	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	259,200	383,039	1.4778
17 Mar 2014 14:51:19	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	233,280	344,989	1.4789
13 Mar 2014 18:45:35	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	207,360	305,386	1.4727
12 Mar 2014 16:10:25	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	181,440	267,246	1.4729
11 Mar 2014 13:45:39	1294522	16294756	hadcm3n_866a_1980_40_008514505_0	155,520	231,484	1.4885