Task 15282441

Name	hadcm3n_zk78_1880_40_008201139_2
Workunit	8356263
Created	13 Sep 2012, 14:25:53 UTC
Sent	13 Sep 2012, 14:33:17 UTC
Report deadline	13 Dec 2012, 22:00:28 UTC
Received	3 Oct 2012, 19:07:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1181449
Run time	11 days 13 hours 3 min 28 sec
CPU time	11 days 3 hours 33 min 54 sec
Validate state	Invalid
Credit	8,087.04
Device peak FLOPS	3.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5140, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5140, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5140, iMonCtr=1 Model crash detected, will try to restart... 07:04:16 (10732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:54:52 (5200): No heartbeat from core client for 30 sec - exiting 08:54:53 (5200): No heartbeat from core client for 30 sec - exiting 08:54:54 (5200): No heartbeat from core client for 30 sec - exiting 08:54:55 (5200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5640, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Sep 2012 21:27:51	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	673,920	936,767	1.3900
30 Sep 2012 10:11:49	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	648,000	900,014	1.3889
29 Sep 2012 15:02:11	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	622,080	863,460	1.3880
28 Sep 2012 21:02:50	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	596,160	826,994	1.3872
27 Sep 2012 18:38:32	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	570,240	790,468	1.3862
27 Sep 2012 08:39:06	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	544,320	754,071	1.3853
26 Sep 2012 16:16:06	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	518,400	719,147	1.3872
26 Sep 2012 03:16:23	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	492,480	684,724	1.3904
25 Sep 2012 16:47:58	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	466,560	648,321	1.3896
25 Sep 2012 05:37:18	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	440,640	611,898	1.3887
24 Sep 2012 19:22:45	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	414,720	575,580	1.3879
23 Sep 2012 21:52:05	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	388,800	539,298	1.3871
22 Sep 2012 21:53:22	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	362,880	502,829	1.3857
22 Sep 2012 11:18:26	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	336,960	466,858	1.3855
22 Sep 2012 01:36:24	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	311,040	431,128	1.3861
21 Sep 2012 12:13:36	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	285,120	395,140	1.3859
20 Sep 2012 14:38:01	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	259,200	358,998	1.3850
20 Sep 2012 04:05:33	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	233,280	322,922	1.3843
19 Sep 2012 17:22:29	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	207,360	286,960	1.3839
19 Sep 2012 06:47:39	1181449	15282441	hadcm3n_zk78_1880_40_008201139_2	181,440	250,760	1.3821