Task 13103047

Name	hadcm3n_ycws_1900_40_007349590_0
Workunit	7547020
Created	6 Jul 2011, 13:59:35 UTC
Sent	17 Jul 2011, 13:55:53 UTC
Report deadline	16 Oct 2011, 21:23:04 UTC
Received	23 Sep 2011, 11:15:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1022824
Run time	47 days 14 hours 34 min 12 sec
CPU time	31 days 2 hours 47 min 17 sec
Validate state	Invalid
Credit	10,575.36
Device peak FLOPS	1.36 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:26:49 (6552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:30:37 (7084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:30:38 (7084): No heartbeat from core client for 30 sec - exiting 07:30:39 (7084): No heartbeat from core client for 30 sec - exiting 07:30:40 (7084): No heartbeat from core client for 30 sec - exiting 07:33:06 (8152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:33:07 (8152): No heartbeat from core client for 30 sec - exiting 07:33:08 (8152): No heartbeat from core client for 30 sec - exiting 07:33:09 (8152): No heartbeat from core client for 30 sec - exiting 07:33:10 (8152): No heartbeat from core client for 30 sec - exiting 07:35:08 (2144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:35:09 (2144): No heartbeat from core client for 30 sec - exiting 07:35:10 (2144): No heartbeat from core client for 30 sec - exiting 07:35:11 (2144): No heartbeat from core client for 30 sec - exiting 07:35:12 (2144): No heartbeat from core client for 30 sec - exiting 07:37:15 (6628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:16 (6628): No heartbeat from core client for 30 sec - exiting 07:39:20 (6800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:41:17 (7460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:41:18 (7460): No heartbeat from core client for 30 sec - exiting 07:43:26 (7372): No heartbeat from core client for 30 sec - exiting 07:43:27 (7372): No heartbeat from core client for 30 sec - exiting 07:43:28 (7372): No heartbeat from core client for 30 sec - exiting 07:43:29 (7372): No heartbeat from core client for 30 sec - exiting 07:43:30 (7372): No heartbeat from core client for 30 sec - exiting 07:43:31 (7372): No heartbeat from core client for 30 sec - exiting 07:43:32 (7372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:45:37 (6552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:45:38 (6552): No heartbeat from core client for 30 sec - exiting 07:45:39 (6552): No heartbeat from core client for 30 sec - exiting 07:45:40 (6552): No heartbeat from core client for 30 sec - exiting 07:45:41 (6552): No heartbeat from core client for 30 sec - exiting 07:45:42 (6552): No heartbeat from core client for 30 sec - exiting 07:51:51 (5816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:51:52 (5816): No heartbeat from core client for 30 sec - exiting 07:51:53 (5816): No heartbeat from core client for 30 sec - exiting 07:51:54 (5816): No heartbeat from core client for 30 sec - exiting 07:51:55 (5816): No heartbeat from core client for 30 sec - exiting 07:51:56 (5816): No heartbeat from core client for 30 sec - exiting 07:51:57 (5816): No heartbeat from core client for 30 sec - exiting 07:52:09 (2816): Can't acquire lockfile (32) - waiting 35s 07:54:32 (2816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:54:33 (2816): No heartbeat from core client for 30 sec - exiting 07:54:35 (2816): No heartbeat from core client for 30 sec - exiting 07:54:36 (2816): No heartbeat from core client for 30 sec - exiting 07:58:22 (6920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:58:23 (6920): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7532, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6960, iMonCtr=1 Model crash deCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is nCPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on ycwsko.dab78f0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4376, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5064, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:54:16 (4800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:54:17 (4800): No heartbeat from core client for 30 sec - exiting 11:47:13 (960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:47:14 (960): No heartbeat from core client for 30 sec - exiting 11:47:15 (960): No heartbeat from core client for 30 sec - exiting 11:47:16 (960): No heartbeat from core client for 30 sec - exiting 11:47:17 (960): No heartbeat from core client for 30 sec - exiting 11:47:18 (960): No heartbeat from core client for 30 sec - exiting 11:47:19 (960): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3872, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:11:24 (1456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:25 (1456): No heartbeat from core client for 30 sec - exiting 18:11:26 (1456): No heartbeat from core client for 30 sec - exiting 18:11:27 (1456): No heartbeat from core client for 30 sec - exiting 18:11:28 (1456): No heartbeat from core client for 30 sec - exiting 18:11:29 (1456): No heartbeat from core client for 30 sec - exiting 18:11:30 (1456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:27:46 (2700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:27:47 (2700): No heartbeat from core client for 30 sec - exiting 11:27:48 (2700): No heartbeat from core client for 30 sec - exiting 11:27:49 (2700): No heartbeat from core client for 30 sec - exiting 11:27:50 (2700): No heartbeat from core client for 30 sec - exiting 11:27:51 (2700): No heartbeat from core client for 30 sec - exiting 11:27:52 (2700): No heartbeat from core client for 30 sec - exiting 11:27:53 (2700): No heartbeat from core client for 30 sec - exiting 11:27:54 (2700): No heartbeat from core client for 30 sec - exiting 11:27:55 (2700): No heartbeat from core client for 30 sec - exiting 11:27:56 (2700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7332, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7332, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7332, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7332, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7332, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7332, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Sep 2011 02:52:48	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	881,280	2,620,282	2.9733
19 Sep 2011 17:45:56	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	855,360	2,545,086	2.9755
10 Sep 2011 02:26:13	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	829,440	2,471,719	2.9800
05 Sep 2011 15:10:25	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	803,520	2,397,421	2.9836
04 Sep 2011 10:31:10	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	777,600	2,316,559	2.9791
03 Sep 2011 09:36:44	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	751,680	2,237,263	2.9764
02 Sep 2011 09:05:10	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	725,760	2,157,156	2.9723
01 Sep 2011 03:59:41	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	699,840	2,077,683	2.9688
30 Aug 2011 05:45:34	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	673,920	1,997,409	2.9639
29 Aug 2011 02:17:54	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	648,000	1,917,707	2.9594
27 Aug 2011 22:11:13	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	622,080	1,836,211	2.9517
26 Aug 2011 20:14:32	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	596,160	1,758,659	2.9500
25 Aug 2011 04:15:46	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	570,240	1,680,808	2.9475
21 Aug 2011 13:04:39	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	544,320	1,602,278	2.9436
19 Aug 2011 08:59:18	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	518,400	1,533,680	2.9585
17 Aug 2011 04:46:21	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	492,480	1,463,453	2.9716
11 Aug 2011 13:33:59	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	466,560	1,393,569	2.9869
09 Aug 2011 20:37:58	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	440,640	1,324,175	3.0051
06 Aug 2011 19:07:53	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	414,720	1,252,450	3.0200
03 Aug 2011 11:29:20	1022824	13103047	hadcm3n_ycws_1900_40_007349590_0	388,800	1,181,152	3.0379