Task 14294812

Name	hadcm3n_yhzo_1940_40_007834035_2
Workunit	7989147
Created	20 Mar 2012, 1:13:04 UTC
Sent	20 Mar 2012, 1:31:50 UTC
Report deadline	19 Jun 2012, 8:59:01 UTC
Received	30 Mar 2012, 12:01:33 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1146627
Run time	9 days 15 hours 8 min 19 sec
CPU time	2 days 14 hours 35 min 54 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	3.10 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.8.13</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:26:24 (4968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:34:18 (6196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:20 (6196): No heartbeat from core client for 30 sec - exiting 22:49:14 (10116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:49:15 (10116): No heartbeat from core client for 30 sec - exiting 22:49:16 (10116): No heartbeat from core client for 30 sec - exiting 23:19:42 (10152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:44 (10152): No heartbeat from core client for 30 sec - exiting 23:29:48 (10612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:29:49 (10612): No heartbeat from core client for 30 sec - exiting 23:33:49 (6524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:33:59 (6524): No heartbeat from core client for 30 sec - exiting 23:34:00 (6524): No heartbeat from core client for 30 sec - exiting 23:52:25 (3156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:52:27 (3156): No heartbeat from core client for 30 sec - exiting 23:52:38 (3156): No heartbeat from core client for 30 sec - exiting 23:52:39 (3156): No heartbeat from core client for 30 sec - exiting 23:52:40 (3156): No heartbeat from core client for 30 sec - exiting 23:52:41 (3156): No heartbeat from core client for 30 sec - exiting 23:52:42 (3156): No heartbeat from core client for 30 sec - exiting 23:52:43 (3156): No heartbeat from core client for 30 sec - exiting 23:52:44 (3156): No heartbeat from core client for 30 sec - exiting 23:54:08 (2208): No heartbeat from core client for 30 sec - exiting 23:54:09 (2208): No heartbeat from core client for 30 sec - exiting 23:54:10 (2208): No heartbeat from core client for 30 sec - exiting 23:54:12 (2208): No heartbeat from core client for 30 sec - exiting 23:54:13 (2208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:39:16 (9928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:39:19 (9928): No heartbeat from core client for 30 sec - exiting 00:46:55 (11684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:46:58 (11684): No heartbeat from core client for 30 sec - exiting 00:46:59 (11684): No heartbeat from core client for 30 sec - exiting 00:47:00 (11684): No heartbeat from core client for 30 sec - exiting 00:55:29 (9496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:55:52 (11976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:55:51 (11836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:55:53 (11836): No heartbeat from core client for 30 sec - exiting 03:55:54 (11836): No heartbeat from core client for 30 sec - exiting 03:55:55 (11836): No heartbeat from core client for 30 sec - exiting 03:55:56 (11836): No heartbeat from core client for 30 sec - exiting 03:55:57 (11836): No heartbeat from core client for 30 sec - exiting 04:56:13 (11492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:56:15 (11492): No heartbeat from core client for 30 sec - exiting 05:56:19 (3304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:56:20 (3304): No heartbeat from core client for 30 sec - exiting 05:56:21 (3304): No heartbeat from core client for 30 sec - exiting 09:57:41 (5464): No heartbeat from core client for 30 sec - exiting 09:57:42 (5464): No heartbeat from core client for 30 sec - exiting 09:57:45 (5464): No heartbeat from core client for 30 sec - exiting 09:57:47 (5464): No heartbeat from core client for 30 sec - exiting 09:57:48 (5464): No heartbeat from core client for 30 sec - exiting 09:57:49 (5464): No heartbeat from core client for 30 sec - exiting 09:57:50 (5464): No heartbeat from core client for 30 sec - exiting 09:57:51 (5464): No heartbeat from core client for 30 sec - exiting 09:57:52 (5464): No heartbeat from core client for 30 sec - exiting 09:57:53 (5464): No heartbeat from core client for 30 sec - exiting 09:57:56 (5464): No heartbeat from core client for 30 sec - exiting 09:57:57 (5464): No heartbeat from core client for 30 sec - exiting 09:57:58 (5464): No heartbeat from core client for 30 sec - exiting 09:58:04 (5464): No heartbeat from core client for 30 sec - exiting 09:58:05 (5464): No heartbeat from core client for 30 sec - exiting 09:58:07 (5464): No heartbeat from core client for 30 sec - exiting 09:58:08 (5464): No heartbeat from core client for 30 sec - exiting 09:58:09 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:58:10 (5464): No heartbeat from core client for 30 sec - exiting 09:59:46 (7424): No heartbeat from core client for 30 sec - exiting 09:59:47 (7424): No heartbeat from core client for 30 sec - exiting 09:59:48 (7424): No heartbeat from core client for 30 sec - exiting 09:59:50 (7424): No heartbeat from core client for 30 sec - exiting 09:59:51 (7424): No heartbeat from core client for 30 sec - exiting 09:59:52 (7424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:21:05 (12084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:21:07 (12084): No heartbeat from core client for 30 sec - exiting 10:21:08 (12084): No heartbeat from core client for 30 sec - exiting 10:21:10 (12084): No heartbeat from core client for 30 sec - exiting 11:00:12 (11988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:00:13 (11988): No heartbeat from core client for 30 sec - exiting 11:00:15 (11988): No heartbeat from core client for 30 sec - exiting 11:00:16 (11988): No heartbeat from core client for 30 sec - exiting 11:00:17 (11988): No heartbeat from core client for 30 sec - exiting 11:00:18 (11988): No heartbeat from core client for 30 sec - exiting 11:00:19 (11988): No heartbeat from core client for 30 sec - exiting 11:00:20 (11988): No heartbeat from core client for 30 sec - exiting 12:00:38 (11524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:00:44 (11524): No heartbeat from core client for 30 sec - exiting 12:00:45 (11524): No heartbeat from core client for 30 sec - exiting 12:02:28 (4980): No heartbeat from core client for 30 sec - exiting 12:02:29 (4980): No heartbeat from core client for 30 sec - exiting 12:02:30 (4980): No heartbeat from core client for 30 sec - exiting 12:02:31 (4980): No heartbeat from core client for 30 sec - exiting 12:02:32 (4980): No heartbeat from core client for 30 sec - exiting 12:02:34 (4980): No heartbeat from core client for 30 sec - exiting 12:02:35 (4980): No heartbeat from core client for 30 sec - exiting 12:02:36 (4980): No heartbeat from core client for 30 sec - exiting 12:02:37 (4980): No heartbeat from core client for 30 sec - exiting 12:02:38 (4980): No heartbeat from core client for 30 sec - exiting 12:02:39 (4980): No heartbeat from core client for 30 sec - exiting 12:02:40 (4980): No heartbeat from core client for 30 sec - exiting 12:02:41 (4980): No heartbeat from core client for 30 sec - exiting 12:02:42 (4980): No heartbeat from core client for 30 sec - exiting 12:02:43 (4980): No heartbeat from core client for 30 sec - exiting 12:02:44 (4980): No heartbeat from core client for 30 sec - exiting 12:02:46 (4980): No heartbeat from core client for 30 sec - exiting 12:02:47 (4980): No heartbeat from core client for 30 sec - exiting 12:02:48 (4980): No heartbeat from core client for 30 sec - exiting 12:02:49 (4980): No heartbeat from core client for 30 sec - exiting 12:02:50 (4980): No heartbeat from core client for 30 sec - exiting 12:02:52 (4980): No heartbeat from core client for 30 sec - exiting 12:02:53 (4980): No heartbeat from core client for 30 sec - exiting 12:02:54 (4980): No heartbeat from core client for 30 sec - exiting 12:02:55 (4980): No heartbeat from core client for 30 sec - exiting 12:02:56 (4980): No heartbeat from core client for 30 sec - exiting 12:02:57 (4980): No heartbeat from core client for 30 sec - exiting 12:02:58 (4980): No heartbeat from core client for 30 sec - exiting 12:02:59 (4980): No heartbeat from core client for 30 sec - exiting 12:03:00 (4980): No heartbeat from core client for 30 sec - exiting 12:03:01 (4980): No heartbeat from core client for 30 sec - exiting 12:03:02 (4980): No heartbeat from core client for 30 sec - exiting 12:03:04 (4980): No heartbeat from core client for 30 sec - exiting 12:03:05 (4980): No heartbeat from core client for 30 sec - exiting 12:03:06 (4980): No heartbeat from core client for 30 sec - exiting 12:03:07 (4980): No heartbeat from core client for 30 sec - exiting 12:03:08 (4980): No heartbeat from core client for 30 sec - exiting 12:03:09 (4980): No heartbeat from core client for 30 sec - exiting 12:03:10 (4980): No heartbeat from core client for 30 sec - exiting 12:03:11 (4980): No heartbeat from core client for 30 sec - exiting 12:03:12 (4980): No heartbeat from core client for 30 sec - exiting 12:03:13 (4980): No heartbeat from core client for 30 sec - exiting 12:03:14 (4980): No heartbeat from core client for 30 sec - exiting 12:03:16 (4980): No heartbeat from core client for 30 sec - exiting 12:03:17 (4980): No heartbeat from core client for 30 sec - exiting 12:03:18 (4980): No heartbeat from core client for 30 sec - exiting 12:03:19 (4980): No heartbeat from core client for 30 sec - exiting 12:03:20 (4980): No heartbeat from core client for 30 sec - exiting 12:03:21 (4980): No heartbeat from core client for 30 sec - exiting 12:03:23 (4980): No heartbeat from core client for 30 sec - exiting 12:03:24 (4980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:10:46 (9364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:00:59 (7548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:01:00 (7548): No heartbeat from core client for 30 sec - exiting 13:01:01 (7548): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10500, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Mar 2012 04:57:17	1146627	14294812	hadcm3n_yhzo_1940_40_007834035_2	181,440	220,687	1.2163
27 Mar 2012 15:47:50	1146627	14294812	hadcm3n_yhzo_1940_40_007834035_2	155,520	190,147	1.2227
25 Mar 2012 07:43:26	1146627	14294812	hadcm3n_yhzo_1940_40_007834035_2	129,600	159,277	1.2290
22 Mar 2012 18:07:40	1146627	14294812	hadcm3n_yhzo_1940_40_007834035_2	103,680	128,314	1.2376
21 Mar 2012 19:27:14	1146627	14294812	hadcm3n_yhzo_1940_40_007834035_2	77,760	96,434	1.2401
21 Mar 2012 03:54:32	1146627	14294812	hadcm3n_yhzo_1940_40_007834035_2	51,840	64,535	1.2449
20 Mar 2012 19:09:28	1146627	14294812	hadcm3n_yhzo_1940_40_007834035_2	25,920	32,287	1.2456