Task 15657779

Name	hadcm3n_zn4o_1960_40_008326701_1
Workunit	8477836
Created	11 Mar 2013, 10:10:47 UTC
Sent	11 Mar 2013, 10:11:18 UTC
Report deadline	10 Jun 2013, 17:38:29 UTC
Received	12 Apr 2013, 9:00:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1181449
Run time	15 days 4 hours 26 min 31 sec
CPU time	14 days 14 hours 4 min 20 sec
Validate state	Invalid
Credit	10,264.32
Device peak FLOPS	3.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3460, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3460, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3460, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4828, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:23:42 (6756): No heartbeat from core client for 30 sec - exiting 14:23:43 (6756): No heartbeat from core client for 30 sec - exiting 14:23:44 (6756): No heartbeat from core client for 30 sec - exiting 14:23:45 (6756): No heartbeat from core client for 30 sec - exiting 14:23:46 (6756): No heartbeat from core client for 30 sec - exiting 14:23:47 (6756): No heartbeat from core client for 30 sec - exiting 14:23:49 (6756): No heartbeat from core client for 30 sec - exiting 14:23:50 (6756): No heartbeat from core client for 30 sec - exiting 14:23:51 (6756): No heartbeat from core client for 30 sec - exiting 14:23:52 (6756): No heartbeat from core client for 30 sec - exiting 14:23:53 (6756): No heartbeat from core client for 30 sec - exiting 14:23:54 (6756): No heartbeat from core client for 30 sec - exiting 14:23:55 (6756): No heartbeat from core client for 30 sec - exiting 14:23:56 (6756): No heartbeat from core client for 30 sec - exiting 14:23:57 (6756): No heartbeat from core client for 30 sec - exiting 14:23:58 (6756): No heartbeat from core client for 30 sec - exiting 14:23:59 (6756): No heartbeat from core client for 30 sec - exiting 14:24:01 (6756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2200, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:46:11 (3724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:13:14 (4104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:22:06 (5552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5072, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4928, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6804, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5016, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3312, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3312, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3312, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6164, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6652, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5208, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Apr 2013 03:06:02	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	855,360	1,226,426	1.4338
06 Apr 2013 16:21:09	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	829,440	1,189,248	1.4338
06 Apr 2013 05:27:19	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	803,520	1,150,346	1.4316
05 Apr 2013 17:07:41	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	777,600	1,111,604	1.4295
05 Apr 2013 05:58:28	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	751,680	1,072,663	1.4270
04 Apr 2013 18:41:03	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	725,760	1,033,558	1.4241
04 Apr 2013 07:40:12	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	699,840	994,625	1.4212
03 Apr 2013 20:14:16	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	673,920	955,776	1.4182
03 Apr 2013 08:55:37	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	648,000	916,900	1.4150
02 Apr 2013 22:06:07	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	622,080	877,856	1.4112
02 Apr 2013 10:34:19	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	596,160	839,268	1.4078
01 Apr 2013 13:16:05	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	570,240	801,467	1.4055
28 Mar 2013 22:25:32	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	544,320	765,608	1.4065
28 Mar 2013 11:24:49	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	518,400	727,534	1.4034
27 Mar 2013 14:30:27	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	492,480	689,094	1.3992
25 Mar 2013 17:17:29	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	466,560	650,646	1.3946
23 Mar 2013 22:18:15	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	440,640	612,801	1.3907
23 Mar 2013 11:14:16	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	414,720	574,154	1.3844
22 Mar 2013 20:04:44	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	388,800	535,791	1.3781
22 Mar 2013 09:24:07	1181449	15657779	hadcm3n_zn4o_1960_40_008326701_1	362,880	497,238	1.3703