Task 13354391

Name	hadcm3n_o19i_1940_40_007444864_2
Workunit	7642367
Created	9 Sep 2011, 12:43:50 UTC
Sent	18 Sep 2011, 6:31:45 UTC
Report deadline	18 Dec 2011, 13:58:56 UTC
Received	26 Sep 2011, 8:25:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1103559
Run time	7 days 15 hours 44 min 5 sec
CPU time	5 days 10 hours 37 min 23 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	1.97 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:11:58 (7844): No heartbeat from core client for 30 sec - exiting 06:11:59 (7844): No heartbeat from core client for 30 sec - exiting 06:12:00 (7844): No heartbeat from core client for 30 sec - exiting 06:12:01 (7844): No heartbeat from core client for 30 sec - exiting 06:12:02 (7844): No heartbeat from core client for 30 sec - exiting 06:12:03 (7844): No heartbeat from core client for 30 sec - exiting 06:12:04 (7844): No heartbeat from core client for 30 sec - exiting 06:12:05 (7844): No heartbeat from core client for 30 sec - exiting 06:12:06 (7844): No heartbeat from core client for 30 sec - exiting 06:12:07 (7844): No heartbeat from core client for 30 sec - exiting 06:12:08 (7844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:39:53 (4716): No heartbeat from core client for 30 sec - exiting 17:39:54 (4716): No heartbeat from core client for 30 sec - exiting 17:39:55 (4716): No heartbeat from core client for 30 sec - exiting 17:39:56 (4716): No heartbeat from core client for 30 sec - exiting 17:39:57 (4716): No heartbeat from core client for 30 sec - exiting 17:39:58 (4716): No heartbeat from core client for 30 sec - exiting 17:39:59 (4716): No heartbeat from core client for 30 sec - exiting 17:40:00 (4716): No heartbeat from core client for 30 sec - exiting 17:40:01 (4716): No heartbeat from core client for 30 sec - exiting 17:40:02 (4716): No heartbeat from core client for 30 sec - exiting 17:40:03 (4716): No heartbeat from core client for 30 sec - exiting 17:40:04 (4716): No heartbeat from core client for 30 sec - exiting 17:40:05 (4716): No heartbeat from core client for 30 sec - exiting 17:40:06 (4716): No heartbeat from core client for 30 sec - exiting 17:40:07 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:28:03 (6148): No heartbeat from core client for 30 sec - exiting 06:28:04 (6148): No heartbeat from core client for 30 sec - exiting 06:28:05 (6148): No heartbeat from core client for 30 sec - exiting 06:28:06 (6148): No heartbeat from core client for 30 sec - exiting 06:28:07 (6148): No heartbeat from core client for 30 sec - exiting 06:28:08 (6148): No heartbeat from core client for 30 sec - exiting 06:28:09 (6148): No heartbeat from core client for 30 sec - exiting 06:28:10 (6148): No heartbeat from core client for 30 sec - exiting 06:28:11 (6148): No heartbeat from core client for 30 sec - exiting 06:28:12 (6148): No heartbeat from core client for 30 sec - exiting 06:28:13 (6148): No heartbeat from core client for 30 sec - exiting 06:28:14 (6148): No heartbeat from core client for 30 sec - exiting 06:28:15 (6148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6240, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4780, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4780, iMonCtr=1 Model crash detected, will try to restart... 06:15:23 (4500): No heartbeat from core client for 30 sec - exiting 06:15:28 (4500): No heartbeat from core client for 30 sec - exiting 06:15:30 (4500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:27:58 (7536): No heartbeat from core client for 30 sec - exiting 06:27:59 (7536): No heartbeat from core client for 30 sec - exiting 06:28:00 (7536): No heartbeat from core client for 30 sec - exiting 06:28:01 (7536): No heartbeat from core client for 30 sec - exiting 06:28:02 (7536): No heartbeat from core client for 30 sec - exiting 06:28:03 (7536): No heartbeat from core client for 30 sec - exiting 06:28:04 (7536): No heartbeat from core client for 30 sec - exiting 06:28:05 (7536): No heartbeat from core client for 30 sec - exiting 06:28:06 (7536): No heartbeat from core client for 30 sec - exiting 06:28:07 (7536): No heartbeat from core client for 30 sec - exiting 06:28:08 (7536): No heartbeat from core client for 30 sec - exiting 06:28:09 (7536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:22 (3584): No heartbeat from core client for 30 sec - exiting 18:18:23 (3584): No heartbeat from core client for 30 sec - exiting 18:18:24 (3584): No heartbeat from core client for 30 sec - exiting 18:18:25 (3584): No heartbeat from core client for 30 sec - exiting 18:18:26 (3584): No heartbeat from core client for 30 sec - exiting 18:18:27 (3584): No heartbeat from core client for 30 sec - exiting 18:18:28 (3584): No heartbeat from core client for 30 sec - exiting 18:18:29 (3584): No heartbeat from core client for 30 sec - exiting 18:18:30 (3584): No heartbeat from core client for 30 sec - exiting 18:18:31 (3584): No heartbeat from core client for 30 sec - exiting 18:18:32 (3584): No heartbeat from core client for 30 sec - exiting 18:18:33 (3584): No heartbeat from core client for 30 sec - exiting 18:18:34 (3584): No heartbeat from core client for 30 sec - exiting 18:18:35 (3584): No heartbeat from core client for 30 sec - exiting 18:18:36 (3584): No heartbeat from core client for 30 sec - exiting 18:18:37 (3584): No heartbeat from core client for 30 sec - exiting 18:18:38 (3584): No heartbeat from core client for 30 sec - exiting 18:18:39 (3584): No heartbeat from core client for 30 sec - exiting 18:18:40 (3584): No heartbeat from core client for 30 sec - exiting 18:18:41 (3584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:33:50 (7652): No heartbeat from core client for 30 sec - exiting 06:33:51 (7652): No heartbeat from core client for 30 sec - exiting 06:33:52 (7652): No heartbeat from core client for 30 sec - exiting 06:33:53 (7652): No heartbeat from core client for 30 sec - exiting 06:33:54 (7652): No heartbeat from core client for 30 sec - exiting 06:33:55 (7652): No heartbeat from core client for 30 sec - exiting 06:33:56 (7652): No heartbeat from core client for 30 sec - exiting 06:33:57 (7652): No heartbeat from core client for 30 sec - exiting 06:33:58 (7652): No heartbeat from core client for 30 sec - exiting 06:33:59 (7652): No heartbeat from core client for 30 sec - exiting 06:34:00 (7652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:35:47 (5492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7224, iMonCtr=1 Model crash detected, will try to restart... 07:56:12 (7020): No heartbeat from core client for 30 sec - exiting 07:56:13 (7020): No heartbeat from core client for 30 sec - exiting 07:56:14 (7020): No heartbeat from core client for 30 sec - exiting 07:56:15 (7020): No heartbeat from core client for 30 sec - exiting 07:56:16 (7020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:03:18 (4976): No heartbeat from core client for 30 sec - exiting 19:03:20 (4976): No heartbeat from core client for 30 sec - exiting 19:03:21 (4976): No heartbeat from core client for 30 sec - exiting 19:03:22 (4976): No heartbeat from core client for 30 sec - exiting 19:03:23 (4976): No heartbeat from core client for 30 sec - exiting 19:03:24 (4976): No heartbeat from core client for 30 sec - exiting 19:03:25 (4976): No heartbeat from core client for 30 sec - exiting 19:03:26 (4976): No heartbeat from core client for 30 sec - exiting 19:03:27 (4976): No heartbeat from core client for 30 sec - exiting 19:03:28 (4976): No heartbeat from core client for 30 sec - exiting 19:03:29 (4976): No heartbeat from core client for 30 sec - exiting 19:03:30 (4976): No heartbeat from core client for 30 sec - exiting 19:03:31 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:39:33 (6660): No heartbeat from core client for 30 sec - exiting 01:39:34 (6660): No heartbeat from core client for 30 sec - exiting 01:39:35 (6660): No heartbeat from core client for 30 sec - exiting 01:39:36 (6660): No heartbeat from core client for 30 sec - exiting 01:39:37 (6660): No heartbeat from core client for 30 sec - exiting 01:39:38 (6660): No heartbeat from core client for 30 sec - exiting 01:39:39 (6660): No heartbeat from core client for 30 sec - exiting 01:39:40 (6660): No heartbeat from core client for 30 sec - exiting 01:39:41 (6660): No heartbeat from core client for 30 sec - exiting 01:39:42 (6660): No heartbeat from core client for 30 sec - exiting 01:39:43 (6660): No heartbeat from core client for 30 sec - exiting 01:39:44 (6660): No heartbeat from core client for 30 sec - exiting 01:39:45 (6660): No heartbeat from core client for 30 sec - exiting 01:39:46 (6660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:55:52 (5892): No heartbeat from core client for 30 sec - exiting 07:55:53 (5892): No heartbeat from core client for 30 sec - exiting 07:55:54 (5892): No heartbeat from core client for 30 sec - exiting 07:55:55 (5892): No heartbeat from core client for 30 sec - exiting 07:55:56 (5892): No heartbeat from core client for 30 sec - exiting 07:55:57 (5892): No heartbeat from core client for 30 sec - exiting 07:55:58 (5892): No heartbeat from core client for 30 sec - exiting 07:55:59 (5892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4508, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4508, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4508, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4508, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4508, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4508, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Sep 2011 11:02:09	1103559	13354391	hadcm3n_o19i_1940_40_007444864_2	207,360	424,354	2.0465
24 Sep 2011 10:52:44	1103559	13354391	hadcm3n_o19i_1940_40_007444864_2	181,440	371,157	2.0456
23 Sep 2011 13:39:35	1103559	13354391	hadcm3n_o19i_1940_40_007444864_2	155,520	318,400	2.0473
22 Sep 2011 15:44:36	1103559	13354391	hadcm3n_o19i_1940_40_007444864_2	129,600	265,149	2.0459
21 Sep 2011 21:05:05	1103559	13354391	hadcm3n_o19i_1940_40_007444864_2	103,680	211,969	2.0445
20 Sep 2011 22:14:37	1103559	13354391	hadcm3n_o19i_1940_40_007444864_2	77,760	158,797	2.0421
20 Sep 2011 01:17:08	1103559	13354391	hadcm3n_o19i_1940_40_007444864_2	51,840	105,983	2.0444
19 Sep 2011 04:15:12	1103559	13354391	hadcm3n_o19i_1940_40_007444864_2	25,920	52,789	2.0366