Task 13639339

Name	hadcm3n_o46t_1940_40_007542996_4
Workunit	7740228
Created	16 Nov 2011, 15:53:40 UTC
Sent	16 Nov 2011, 15:56:17 UTC
Report deadline	15 Feb 2012, 23:23:28 UTC
Received	30 Dec 2011, 13:43:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1096749
Run time	28 days 15 hours 7 min 43 sec
CPU time	18 days 7 hours 38 min 36 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:58:56 (9128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:05:02 (3672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13764, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:59:12 (4536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:20 (6416): Can't acquire lockfile (32) - waiting 35s 20:05:17 (6416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:14:26 (4792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:17:28 (7004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:46 (6632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:15 (6200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:56:28 (8172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:31 (7972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:27:08 (8184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:27:09 (8184): No heartbeat from core client for 30 sec - exiting 01:27:10 (8184): No heartbeat from core client for 30 sec - exiting 01:51:21 (9100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:09:28 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:09:35 (3728): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... 23:08:59 (6864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:09:00 (6864): No heartbeat from core client for 30 sec - exiting 23:21:24 (9948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:21:25 (9948): No heartbeat from core client for 30 sec - exiting 23:21:26 (9948): No heartbeat from core client for 30 sec - exiting 23:21:27 (9948): No heartbeat from core client for 30 sec - exiting 23:21:28 (9948): No heartbeat from core client for 30 sec - exiting 23:21:29 (9948): No heartbeat from core client for 30 sec - exiting 23:21:30 (9948): No heartbeat from core client for 30 sec - exiting 23:21:31 (7724): Can't acquire lockfile (32) - waiting 35s 23:30:43 (7724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:39:02 (6728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:39:03 (6728): No heartbeat from core client for 30 sec - exiting 18:54:15 (6264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:54:16 (6264): No heartbeat from core client for 30 sec - exiting 19:18:42 (5580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:18:43 (5580): No heartbeat from core client for 30 sec - exiting 19:18:44 (5580): No heartbeat from core client for 30 sec - exiting 19:18:45 (5580): No heartbeat from core client for 30 sec - exiting 19:18:46 (5580): No heartbeat from core client for 30 sec - exiting 19:18:47 (5580): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6920, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:43:31 (12664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:55:38 (16836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:07:44 (17528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:41:05 (18780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:27:52 (2676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:27:53 (2676): No heartbeat from core client for 30 sec - exiting 14:27:54 (2676): No heartbeat from core client for 30 sec - exiting 14:27:55 (2676): No heartbeat from core client for 30 sec - exiting 14:27:56 (2676): No heartbeat from core client for 30 sec - exiting 14:27:57 (2676): No heartbeat from core client for 30 sec - exiting forrtl: Access is denied. 14:52:09 (2328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:17:11 (7476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:17:12 (7476): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9848, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10532, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7520, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7940, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Dec 2011 21:47:49	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	518,400	1,549,931	2.9898
23 Dec 2011 12:47:09	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	492,480	1,471,839	2.9886
21 Dec 2011 09:28:06	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	466,560	1,394,181	2.9882
18 Dec 2011 16:03:59	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	440,640	1,317,521	2.9900
17 Dec 2011 08:53:58	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	414,720	1,240,790	2.9919
16 Dec 2011 01:09:34	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	388,800	1,162,900	2.9910
13 Dec 2011 21:49:41	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	362,880	1,083,108	2.9848
12 Dec 2011 18:48:22	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	336,960	1,007,530	2.9901
11 Dec 2011 13:07:55	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	311,040	928,817	2.9862
10 Dec 2011 08:26:00	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	285,120	850,693	2.9836
09 Dec 2011 04:16:35	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	259,200	774,277	2.9872
07 Dec 2011 22:04:22	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	233,280	696,533	2.9858
28 Nov 2011 21:16:51	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	207,360	620,288	2.9914
27 Nov 2011 15:17:26	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	181,440	542,483	2.9899
26 Nov 2011 02:22:58	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	155,520	465,707	2.9945
24 Nov 2011 17:03:47	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	129,600	389,256	3.0035
23 Nov 2011 07:02:17	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	103,680	311,992	3.0092
21 Nov 2011 03:37:13	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	77,760	234,178	3.0115
19 Nov 2011 14:20:16	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	51,840	156,584	3.0205
18 Nov 2011 02:42:34	1096749	13639339	hadcm3n_o46t_1940_40_007542996_4	25,920	78,662	3.0348