Task 14278993

Name	hadcm3n_yg8t_1980_40_007831477_1
Workunit	7986589
Created	17 Mar 2012, 18:34:01 UTC
Sent	17 Mar 2012, 18:34:05 UTC
Report deadline	17 Jun 2012, 2:01:16 UTC
Received	24 Apr 2012, 16:30:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1080358
Run time	14 days 5 hours 43 min 56 sec
CPU time	13 days 17 hours 42 min 44 sec
Validate state	Invalid
Credit	8,709.12
Device peak FLOPS	2.30 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:33:03 (40284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:32:41 (47168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:50:31 (6060): No heartbeat from core client for 30 sec - exiting 06:50:32 (6060): No heartbeat from core client for 30 sec - exiting 06:51:04 (6060): No heartbeat from core client for 30 sec - exiting 06:51:05 (6060): No heartbeat from core client for 30 sec - exiting 06:51:06 (6060): No heartbeat from core client for 30 sec - exiting 06:51:07 (6060): No heartbeat from core client for 30 sec - exiting 06:51:08 (6060): No heartbeat from core client for 30 sec - exiting 06:51:09 (6060): No heartbeat from core client for 30 sec - exiting 06:51:10 (6060): No heartbeat from core client for 30 sec - exiting 06:51:11 (6060): No heartbeat from core client for 30 sec - exiting 06:51:12 (6060): No heartbeat from core client for 30 sec - exiting 06:51:13 (6060): No heartbeat from core client for 30 sec - exiting 06:51:14 (6060): No heartbeat from core client for 30 sec - exiting 06:51:15 (6060): No heartbeat from core client for 30 sec - exiting 06:51:16 (6060): No heartbeat from core client for 30 sec - exiting 06:51:17 (6060): No heartbeat from core client for 30 sec - exiting 06:51:18 (6060): No heartbeat from core client for 30 sec - exiting 06:51:19 (6060): No heartbeat from core client for 30 sec - exiting 06:51:20 (6060): No heartbeat from core client for 30 sec - exiting 06:51:21 (6060): No heartbeat from core client for 30 sec - exiting 06:51:22 (6060): No heartbeat from core client for 30 sec - exiting 06:51:23 (6060): No heartbeat from core client for 30 sec - exiting 06:51:24 (6060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:51:25 (6060): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:21:22 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:34:47 (5092): No heartbeat from core client for 30 sec - exiting 06:34:48 (5092): No heartbeat from core client for 30 sec - exiting 06:34:49 (5092): No heartbeat from core client for 30 sec - exiting 06:34:50 (5092): No heartbeat from core client for 30 sec - exiting 06:34:51 (5092): No heartbeat from core client for 30 sec - exiting 06:34:52 (5092): No heartbeat from core client for 30 sec - exiting 06:34:53 (5092): No heartbeat from core client for 30 sec - exiting 06:34:54 (5092): No heartbeat from core client for 30 sec - exiting 06:34:55 (5092): No heartbeat from core client for 30 sec - exiting 06:34:56 (5092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:24:25 (4424): No heartbeat from core client for 30 sec - exiting 19:24:26 (4424): No heartbeat from core client for 30 sec - exiting 19:24:27 (4424): No heartbeat from core client for 30 sec - exiting 19:24:28 (4424): No heartbeat from core client for 30 sec - exiting 19:24:29 (4424): No heartbeat from core client for 30 sec - exiting 19:24:30 (4424): No heartbeat from core client for 30 sec - exiting 19:24:31 (4424): No heartbeat from core client for 30 sec - exiting 19:24:32 (4424): No heartbeat from core client for 30 sec - exiting 19:24:33 (4424): No heartbeat from core client for 30 sec - exiting 19:24:34 (4424): No heartbeat from core client for 30 sec - exiting 19:24:35 (4424): No heartbeat from core client for 30 sec - exiting 19:24:36 (4424): No heartbeat from core client for 30 sec - exiting 19:24:37 (4424): No heartbeat from core client for 30 sec - exiting 19:24:38 (4424): No heartbeat from core client for 30 sec - exiting 19:24:39 (4424): No heartbeat from core client for 30 sec - exiting 19:24:40 (4424): No heartbeat from core client for 30 sec - exiting 19:24:41 (4424): No heartbeat from core client for 30 sec - exiting 19:24:42 (4424): No heartbeat from core client for 30 sec - exiting 19:24:43 (4424): No heartbeat from core client for 30 sec - exiting 19:24:44 (4424): No heartbeat from core client for 30 sec - exiting 19:24:45 (4424): No heartbeat from core client for 30 sec - exiting 19:24:46 (4424): No heartbeat from core client for 30 sec - exiting 19:24:47 (4424): No heartbeat from core client for 30 sec - exiting 19:24:48 (4424): No heartbeat from core client for 30 sec - exiting 19:24:49 (4424): No heartbeat from core client for 30 sec - exiting 19:24:50 (4424): No heartbeat from core client for 30 sec - exiting 19:24:51 (4424): No heartbeat from core client for 30 sec - exiting 19:24:52 (4424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:03:40 (3328): No heartbeat from core client for 30 sec - exiting 18:03:41 (3328): No heartbeat from core client for 30 sec - exiting 18:03:42 (3328): No heartbeat from core client for 30 sec - exiting 18:03:43 (3328): No heartbeat from core client for 30 sec - exiting 18:03:44 (3328): No heartbeat from core client for 30 sec - exiting 18:03:45 (3328): No heartbeat from core client for 30 sec - exiting 18:03:46 (3328): No heartbeat from core client for 30 sec - exiting 18:03:47 (3328): No heartbeat from core client for 30 sec - exiting 18:03:48 (3328): No heartbeat from core client for 30 sec - exiting 18:03:49 (3328): No heartbeat from core client for 30 sec - exiting 18:03:50 (3328): No heartbeat from core client for 30 sec - exiting 18:03:51 (3328): No heartbeat from core client for 30 sec - exiting 18:03:52 (3328): No heartbeat from core client for 30 sec - exiting 18:03:53 (3328): No heartbeat from core client for 30 sec - exiting 18:03:54 (3328): No heartbeat from core client for 30 sec - exiting 18:03:55 (3328): No heartbeat from core client for 30 sec - exiting 18:03:56 (3328): No heartbeat from core client for 30 sec - exiting 18:03:57 (3328): No heartbeat from core client for 30 sec - exiting 18:03:58 (3328): No heartbeat from core client for 30 sec - exiting 18:03:59 (3328): No heartbeat from core client for 30 sec - exiting 18:04:00 (3328): No heartbeat from core client for 30 sec - exiting 18:04:01 (3328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:06:08 (5580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:35 (3056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:46 (3980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:23:59 (4052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:43:14 (4708): No heartbeat from core client for 30 sec - exiting 06:43:15 (4708): No heartbeat from core client for 30 sec - exiting 06:43:16 (4708): No heartbeat from core client for 30 sec - exiting 06:43:17 (4708): No heartbeat from core client for 30 sec - exiting 06:43:18 (4708): No heartbeat from core client for 30 sec - exiting 06:43:19 (4708): No heartbeat from core client for 30 sec - exiting 06:43:20 (4708): No heartbeat from core client for 30 sec - exiting 06:43:21 (4708): No heartbeat from core client for 30 sec - exiting 06:43:22 (4708): No heartbeat from core client for 30 sec - exiting 06:43:23 (4708): No heartbeat from core client for 30 sec - exiting 06:43:24 (4708): No heartbeat from core client for 30 sec - exiting 06:43:25 (4708): No heartbeat from core client for 30 sec - exiting 06:43:26 (4708): No heartbeat from core client for 30 sec - exiting 06:43:27 (4708): No heartbeat from core client for 30 sec - exiting 06:43:28 (4708): No heartbeat from core client for 30 sec - exiting 06:43:29 (4708): No heartbeat from core client for 30 sec - exiting 06:43:30 (4708): No heartbeat from core client for 30 sec - exiting 06:43:31 (4708): No heartbeat from core client for 30 sec - exiting 06:43:32 (4708): No heartbeat from core client for 30 sec - exiting 06:43:33 (4708): No heartbeat from core client for 30 sec - exiting 06:43:34 (4708): No heartbeat from core client for 30 sec - exiting 06:43:35 (4708): No heartbeat from core client for 30 sec - exiting 06:43:36 (4708): No heartbeat from core client for 30 sec - exiting 06:43:37 (4708): No heartbeat from core client for 30 sec - exiting 06:43:38 (4708): No heartbeat from core client for 30 sec - exiting 06:43:39 (4708): No heartbeat from core client for 30 sec - exiting 06:43:40 (4708): No heartbeat from core client for 30 sec - exiting 06:43:41 (4708): No heartbeat from core client for 30 sec - exiting 06:43:42 (4708): No heartbeat from core client for 30 sec - exiting 06:43:43 (4708): No heartbeat from core client for 30 sec - exiting 06:43:44 (4708): No heartbeat from core client for 30 sec - exiting 06:43:45 (4708): No heartbeat from core client for 30 sec - exiting 06:43:46 (4708): No heartbeat from core client for 30 sec - exiting 06:43:47 (4708): No heartbeat from core client for 30 sec - exiting 06:43:48 (4708): No heartbeat from core client for 30 sec - exiting 06:43:49 (4708): No heartbeat from core client for 30 sec - exiting 06:43:50 (4708): No heartbeat from core client for 30 sec - exiting 06:43:51 (4708): No heartbeat from core client for 30 sec - exiting 06:43:52 (4708): No heartbeat from core client for 30 sec - exiting 06:43:53 (4708): No heartbeat from core client for 30 sec - exiting 06:43:54 (4708): No heartbeat from core client for 30 sec - exiting 06:43:55 (4708): No heartbeat from core client for 30 sec - exiting 06:43:56 (4708): No heartbeat from core client for 30 sec - exiting 06:43:57 (4708): No heartbeat from core client for 30 sec - exiting 06:43:58 (4708): No heartbeat from core client for 30 sec - exiting 06:43:59 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5440, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5440, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5440, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5440, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5440, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5440, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Apr 2012 04:00:13	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	725,760	1,176,186	1.6206
23 Apr 2012 12:48:03	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	699,840	1,134,002	1.6204
22 Apr 2012 16:37:18	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	673,920	1,091,376	1.6194
21 Apr 2012 20:13:50	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	648,000	1,049,011	1.6188
21 Apr 2012 04:21:16	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	622,080	1,006,974	1.6187
20 Apr 2012 09:54:36	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	596,160	965,057	1.6188
19 Apr 2012 15:12:28	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	570,240	922,656	1.6180
18 Apr 2012 18:55:15	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	544,320	880,328	1.6173
17 Apr 2012 21:48:25	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	518,400	838,278	1.6170
17 Apr 2012 02:41:47	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	492,480	796,371	1.6171
16 Apr 2012 11:32:23	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	466,560	754,423	1.6170
15 Apr 2012 19:16:49	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	440,640	713,405	1.6190
15 Apr 2012 03:18:35	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	414,720	671,641	1.6195
14 Apr 2012 07:14:43	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	388,800	629,491	1.6191
13 Apr 2012 10:39:00	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	362,880	587,602	1.6193
12 Apr 2012 16:27:54	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	336,960	545,047	1.6175
11 Apr 2012 18:21:45	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	311,040	502,860	1.6167
10 Apr 2012 22:35:15	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	285,120	460,955	1.6167
09 Apr 2012 23:59:18	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	259,200	418,910	1.6162
09 Apr 2012 05:25:43	1080358	14278993	hadcm3n_yg8t_1980_40_007831477_1	233,280	376,458	1.6138