Task 16481357

Name	hadcm3n_oak9_1900_40_008468812_4
Workunit	8619651
Created	5 Apr 2014, 21:28:34 UTC
Sent	5 Apr 2014, 21:30:40 UTC
Report deadline	6 Jul 2014, 4:57:51 UTC
Received	27 Apr 2014, 14:53:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1242215
Run time	10 days 7 hours 2 min 27 sec
CPU time	9 days 7 hours 55 min 44 sec
Validate state	Invalid
Credit	8,709.12
Device peak FLOPS	3.56 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 17:41:44 (6384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:42:54 (7236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:55:18 (4008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:57:02 (5164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:08 (6568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:49 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:51 (5372): No heartbeat from core client for 30 sec - exiting 07:12:52 (5372): No heartbeat from core client for 30 sec - exiting 07:12:53 (5372): No heartbeat from core client for 30 sec - exiting 07:12:54 (5372): No heartbeat from core client for 30 sec - exiting 09:07:43 (6616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:14:09 (4544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:27:24 (1264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:23:05 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:14:20 (4768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:32 (4768): No heartbeat from core client for 30 sec - exiting 19:12:14 (5408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:14:28 (1560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:04:21 (4884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:49:17 (5484): No heartbeat from core client for 30 sec - exiting 07:49:18 (5484): No heartbeat from core client for 30 sec - exiting 07:49:19 (5484): No heartbeat from core client for 30 sec - exiting 07:49:20 (5484): No heartbeat from core client for 30 sec - exiting 07:49:21 (5484): No heartbeat from core client for 30 sec - exiting 07:49:22 (5484): No heartbeat from core client for 30 sec - exiting 07:49:23 (5484): No heartbeat from core client for 30 sec - exiting 07:49:24 (5484): No heartbeat from core client for 30 sec - exiting 07:49:25 (5484): No heartbeat from core client for 30 sec - exiting 07:49:26 (5484): No heartbeat from core client for 30 sec - exiting 07:49:28 (5484): No heartbeat from core client for 30 sec - exiting 07:49:29 (5484): No heartbeat from core client for 30 sec - exiting 07:49:30 (5484): No heartbeat from core client for 30 sec - exiting 07:49:31 (5484): No heartbeat from core client for 30 sec - exiting 07:49:32 (5484): No heartbeat from core client for 30 sec - exiting 07:49:33 (5484): No heartbeat from core client for 30 sec - exiting 07:49:34 (5484): No heartbeat from core client for 30 sec - exiting 07:49:35 (5484): No heartbeat from core client for 30 sec - exiting 07:49:36 (5484): No heartbeat from core client for 30 sec - exiting 07:49:37 (5484): No heartbeat from core client for 30 sec - exiting 07:49:38 (5484): No heartbeat from core client for 30 sec - exiting 07:49:40 (5484): No heartbeat from core client for 30 sec - exiting 07:49:41 (5484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5920, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3308, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Apr 2014 14:57:28	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	725,760	792,841	1.0924
27 Apr 2014 01:02:09	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	699,840	763,546	1.0910
26 Apr 2014 17:24:34	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	673,920	734,529	1.0899
26 Apr 2014 04:27:37	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	648,000	705,508	1.0887
24 Apr 2014 00:53:20	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	622,080	676,354	1.0872
20 Apr 2014 22:13:02	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	596,160	647,244	1.0857
20 Apr 2014 14:05:17	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	570,240	618,038	1.0838
20 Apr 2014 12:23:46	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	544,320	589,344	1.0827
19 Apr 2014 23:21:30	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	518,400	561,353	1.0829
19 Apr 2014 23:21:30	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	492,480	532,988	1.0823
19 Apr 2014 03:38:35	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	466,560	504,429	1.0812
18 Apr 2014 20:57:14	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	440,640	479,115	1.0873
18 Apr 2014 20:57:14	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	414,720	451,536	1.0888
18 Apr 2014 20:57:14	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	388,800	423,038	1.0881
16 Apr 2014 00:14:07	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	362,880	395,096	1.0888
14 Apr 2014 02:19:19	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	336,960	367,439	1.0905
14 Apr 2014 02:19:19	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	311,040	339,778	1.0924
14 Apr 2014 02:19:19	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	285,120	310,616	1.0894
14 Apr 2014 02:19:19	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	259,200	282,051	1.0882
14 Apr 2014 02:19:19	1242215	16481357	hadcm3n_oak9_1900_40_008468812_4	233,280	253,315	1.0859