Task 13606141

Name	hadcm3n_o5wp_1980_40_007538316_0
Workunit	7735548
Created	5 Nov 2011, 20:08:27 UTC
Sent	5 Nov 2011, 21:35:04 UTC
Report deadline	5 Feb 2012, 5:02:15 UTC
Received	11 Jan 2012, 3:48:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	922327
Run time	54 days 9 hours 4 min 10 sec
CPU time	39 days 6 hours 26 min 1 sec
Validate state	Invalid
Credit	10,264.32
Device peak FLOPS	1.72 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:27:23 (9932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:27:25 (9932): No heartbeat from core client for 30 sec - exiting 22:33:26 (13336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:40:05 (6680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:42:59 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:43:00 (5372): No heartbeat from core client for 30 sec - exiting 06:43:01 (5372): No heartbeat from core client for 30 sec - exiting 06:43:02 (5372): No heartbeat from core client for 30 sec - exiting 06:43:03 (5372): No heartbeat from core client for 30 sec - exiting 06:43:04 (5372): No heartbeat from core client for 30 sec - exiting 06:43:05 (5372): No heartbeat from core client for 30 sec - exiting 06:43:06 (5372): No heartbeat from core client for 30 sec - exiting 06:43:07 (5372): No heartbeat from core client for 30 sec - exiting 06:43:08 (5372): No heartbeat from core client for 30 sec - exiting 06:43:09 (5372): No heartbeat from core client for 30 sec - exiting 06:43:10 (5372): No heartbeat from core client for 30 sec - exiting 06:43:11 (5372): No heartbeat from core client for 30 sec - exiting 06:43:12 (5372): No heartbeat from core client for 30 sec - exiting 06:43:13 (5372): No heartbeat from core client for 30 sec - exiting 06:43:14 (5372): No heartbeat from core client for 30 sec - exiting 06:43:15 (5372): No heartbeat from core client for 30 sec - exiting 06:43:16 (5372): No heartbeat from core client for 30 sec - exiting 05:54:27 (13444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:54:28 (13444): No heartbeat from core client for 30 sec - exiting 03:56:58 (8332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:12:35 (332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:36 (332): No heartbeat from core client for 30 sec - exiting 16:50:37 (10008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:39 (10008): No heartbeat from core client for 30 sec - exiting 16:50:40 (10008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:59:04 (6176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:12:59 (3412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:13:01 (3412): No heartbeat from core client for 30 sec - exiting 03:15:31 (5172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:28:22 (960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:28:23 (960): No heartbeat from core client for 30 sec - exiting 15:28:24 (960): No heartbeat from core client for 30 sec - exiting 21:52:29 (4292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1072, iMonCtr=1 Model crash detected, will try to restart... 13:43:59 (2616): No heartbeat from core client for 30 sec - exiting 13:44:00 (2616): No heartbeat from core client for 30 sec - exiting 13:44:02 (2616): No heartbeat from core client for 30 sec - exiting 13:44:03 (2616): No heartbeat from core client for 30 sec - exiting 13:44:04 (2616): No heartbeat from core client for 30 sec - exiting 13:44:05 (2616): No heartbeat from core client for 30 sec - exiting 13:44:06 (2616): No heartbeat from core client for 30 sec - exiting 13:44:07 (2616): No heartbeat from core client for 30 sec - exiting 13:44:08 (2616): No heartbeat from core client for 30 sec - exiting 13:44:09 (2616): No heartbeat from core client for 30 sec - exiting 13:44:10 (2616): No heartbeat from core client for 30 sec - exiting 13:44:11 (2616): No heartbeat from core client for 30 sec - exiting 13:44:12 (2616): No heartbeat from core client for 30 sec - exiting 13:44:13 (2616): No heartbeat from core client for 30 sec - exiting 13:44:14 (2616): No heartbeat from core client for 30 sec - exiting 13:44:15 (2616): No heartbeat from core client for 30 sec - exiting 13:44:16 (2616): No heartbeat from core client for 30 sec - exiting 13:44:17 (2616): No heartbeat from core client for 30 sec - exiting 13:44:18 (2616): No heartbeat from core client for 30 sec - exiting 13:44:19 (2616): No heartbeat from core client for 30 sec - exiting 13:44:20 (2616): No heartbeat from core client for 30 sec - exiting 13:44:21 (2616): No heartbeat from core client for 30 sec - exiting 13:44:22 (2616): No heartbeat from core client for 30 sec - exiting 13:44:23 (2616): No heartbeat from core client for 30 sec - exiting 13:44:24 (2616): No heartbeat from core client for 30 sec - exiting 13:44:25 (2616): No heartbeat from core client for 30 sec - exiting 13:44:26 (2616): No heartbeat from core client for 30 sec - exiting 13:44:27 (2616): No heartbeat from core client for 30 sec - exiting 13:44:28 (2616): No heartbeat from core client for 30 sec - exiting 13:44:29 (2616): No heartbeat from core client for 30 sec - exiting 13:44:30 (2616): No heartbeat from core client for 30 sec - exiting 13:44:31 (2616): No heartbeat from core client for 30 sec - exiting 13:44:32 (2616): No heartbeat from core client for 30 sec - exiting 13:44:33 (2616): No heartbeat from core client for 30 sec - exiting 13:44:34 (2616): No heartbeat from core client for 30 sec - exiting 13:44:35 (2616): No heartbeat from core client for 30 sec - exiting 13:44:36 (2616): No heartbeat from core client for 30 sec - exiting 13:44:37 (2616): No heartbeat from core client for 30 sec - exiting 13:44:38 (2616): No heartbeat from core client for 30 sec - exiting 13:44:39 (2616): No heartbeat from core client for 30 sec - exiting 13:44:40 (2616): No heartbeat from core client for 30 sec - exiting 13:44:41 (2616): No heartbeat from core client for 30 sec - exiting 13:44:42 (2616): No heartbeat from core client for 30 sec - exiting 13:44:43 (2616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:44:45 (2616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 13:47:49 (6052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:07:54 (6484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:07:55 (6484): No heartbeat from core client for 30 sec - exiting 09:07:56 (6484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:01:06 (8104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4732, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:06:41 (3684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:06:42 (3684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jan 2012 02:05:51	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	855,360	3,322,161	3.8839
05 Jan 2012 20:19:23	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	829,440	3,231,768	3.8963
04 Jan 2012 13:52:23	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	803,520	3,139,722	3.9075
02 Jan 2012 23:08:46	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	777,600	3,043,846	3.9144
01 Jan 2012 14:28:39	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	751,680	2,949,852	3.9243
31 Dec 2011 08:56:30	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	725,760	2,858,919	3.9392
30 Dec 2011 02:58:15	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	699,840	2,768,077	3.9553
28 Dec 2011 19:18:33	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	673,920	2,676,566	3.9716
27 Dec 2011 03:41:20	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	648,000	2,579,082	3.9801
24 Dec 2011 23:36:29	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	622,080	2,463,985	3.9609
22 Dec 2011 17:10:48	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	596,160	2,351,787	3.9449
20 Dec 2011 11:04:55	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	570,240	2,239,935	3.9281
18 Dec 2011 02:46:24	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	544,320	2,125,491	3.9049
15 Dec 2011 23:08:59	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	518,400	2,018,614	3.8939
14 Dec 2011 08:42:42	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	492,480	1,926,166	3.9112
11 Dec 2011 06:51:23	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	466,560	1,831,363	3.9252
10 Dec 2011 00:54:09	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	440,640	1,736,543	3.9410
08 Dec 2011 14:39:39	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	414,720	1,640,476	3.9556
05 Dec 2011 18:47:27	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	388,800	1,546,586	3.9778
04 Dec 2011 14:01:14	922327	13606141	hadcm3n_o5wp_1980_40_007538316_0	362,880	1,453,040	4.0042