Task 16481094

Name	hadcm3n_n3yb_1960_40_008409444_4
Workunit	8560300
Created	5 Apr 2014, 16:51:15 UTC
Sent	5 Apr 2014, 16:56:40 UTC
Report deadline	6 Jul 2014, 0:23:51 UTC
Received	27 Apr 2014, 14:53:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1242215
Run time	10 days 11 hours 2 min 9 sec
CPU time	9 days 11 hours 43 min 59 sec
Validate state	Invalid
Credit	8,709.12
Device peak FLOPS	3.56 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 17:41:44 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:42:55 (3616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:55:18 (1956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:57:02 (6596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:08 (6576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:16 (6576): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6408, iMonCtr=1 Model crash detected, will try to restart... 07:12:49 (5364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:51 (5364): No heartbeat from core client for 30 sec - exiting 07:12:52 (5364): No heartbeat from core client for 30 sec - exiting 07:12:53 (5364): No heartbeat from core client for 30 sec - exiting 07:12:54 (5364): No heartbeat from core client for 30 sec - exiting 09:07:44 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:14:10 (6784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:27:24 (5280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:23:06 (5276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:14:20 (4376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:32 (4376): No heartbeat from core client for 30 sec - exiting 08:14:33 (4376): No heartbeat from core client for 30 sec - exiting 08:14:34 (4376): No heartbeat from core client for 30 sec - exiting 08:14:35 (4376): No heartbeat from core client for 30 sec - exiting 08:14:36 (4376): No heartbeat from core client for 30 sec - exiting 08:14:37 (4376): No heartbeat from core client for 30 sec - exiting 08:14:38 (4376): No heartbeat from core client for 30 sec - exiting 08:14:39 (4376): No heartbeat from core client for 30 sec - exiting 08:14:40 (4376): No heartbeat from core client for 30 sec - exiting 08:14:41 (4376): No heartbeat from core client for 30 sec - exiting 08:14:42 (4376): No heartbeat from core client for 30 sec - exiting 19:12:14 (7084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:14:28 (3800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:14:29 (3800): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:04:21 (1420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:49:17 (5508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Apr 2014 14:57:28	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	725,760	794,012	1.0940
27 Apr 2014 01:02:09	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	699,840	764,615	1.0926
26 Apr 2014 12:04:07	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	673,920	735,600	1.0915
25 Apr 2014 22:30:10	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	648,000	706,470	1.0902
21 Apr 2014 22:58:45	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	622,080	677,240	1.0887
20 Apr 2014 22:13:02	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	596,160	647,921	1.0868
20 Apr 2014 12:23:46	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	570,240	618,749	1.0851
20 Apr 2014 03:12:12	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	544,320	590,215	1.0843
19 Apr 2014 23:21:30	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	518,400	562,043	1.0842
19 Apr 2014 23:21:30	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	492,480	533,853	1.0840
18 Apr 2014 20:57:14	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	466,560	505,323	1.0831
18 Apr 2014 20:57:14	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	440,640	480,731	1.0910
18 Apr 2014 20:57:14	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	414,720	452,157	1.0903
17 Apr 2014 01:12:57	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	388,800	423,946	1.0904
14 Apr 2014 02:19:19	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	362,880	396,084	1.0915
14 Apr 2014 02:19:19	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	336,960	367,601	1.0909
14 Apr 2014 02:19:19	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	311,040	339,615	1.0919
14 Apr 2014 02:19:19	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	285,120	310,236	1.0881
14 Apr 2014 02:19:19	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	259,200	282,106	1.0884
14 Apr 2014 02:19:19	1242215	16481094	hadcm3n_n3yb_1960_40_008409444_4	233,280	252,739	1.0834