Task 13374904

Name	hadcm3n_t39h_1940_40_007452362_2
Workunit	7649865
Created	11 Sep 2011, 9:28:10 UTC
Sent	11 Sep 2011, 9:31:58 UTC
Report deadline	11 Dec 2011, 16:59:09 UTC
Received	16 Sep 2011, 1:10:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1122757
Run time	2 days 1 hours 14 min 52 sec
CPU time	2 days 0 hours 26 min 34 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	1.70 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 17:00:43 (5580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:04:34 (2560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:04:40 (2560): No heartbeat from core client for 30 sec - exiting 21:04:41 (2560): No heartbeat from core client for 30 sec - exiting 21:04:42 (2560): No heartbeat from core client for 30 sec - exiting 21:04:43 (2560): No heartbeat from core client for 30 sec - exiting 21:04:44 (2560): No heartbeat from core client for 30 sec - exiting 21:04:45 (2560): No heartbeat from core client for 30 sec - exiting 21:04:46 (2560): No heartbeat from core client for 30 sec - exiting 21:04:47 (2560): No heartbeat from core client for 30 sec - exiting 21:04:48 (2560): No heartbeat from core client for 30 sec - exiting 21:04:50 (2560): No heartbeat from core client for 30 sec - exiting 21:04:51 (2560): No heartbeat from core client for 30 sec - exiting 21:04:52 (2560): No heartbeat from core client for 30 sec - exiting 21:04:53 (2560): No heartbeat from core client for 30 sec - exiting 21:04:54 (2560): No heartbeat from core client for 30 sec - exiting 21:04:55 (2560): No heartbeat from core client for 30 sec - exiting 21:04:56 (2560): No heartbeat from core client for 30 sec - exiting 21:04:57 (2560): No heartbeat from core client for 30 sec - exiting 21:04:58 (2560): No heartbeat from core client for 30 sec - exiting 21:04:59 (2560): No heartbeat from core client for 30 sec - exiting 21:05:00 (2560): No heartbeat from core client for 30 sec - exiting 21:05:02 (2560): No heartbeat from core client for 30 sec - exiting 21:05:03 (2560): No heartbeat from core client for 30 sec - exiting 21:05:04 (2560): No heartbeat from core client for 30 sec - exiting 23:40:03 (5136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:44:13 (6872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:56:54 (7108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:07:08 (2192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:07:55 (2660): No heartbeat from core client for 30 sec - exiting 00:08:04 (2660): No heartbeat from core client for 30 sec - exiting 00:08:05 (2660): No heartbeat from core client for 30 sec - exiting 00:08:07 (2660): No heartbeat from core client for 30 sec - exiting 00:08:08 (2660): No heartbeat from core client for 30 sec - exiting 00:08:09 (2660): No heartbeat from core client for 30 sec - exiting 00:08:10 (2660): No heartbeat from core client for 30 sec - exiting 00:08:11 (2660): No heartbeat from core client for 30 sec - exiting 00:08:12 (2660): No heartbeat from core client for 30 sec - exiting 00:08:13 (2660): No heartbeat from core client for 30 sec - exiting 00:09:21 (2660): No heartbeat from core client for 30 sec - exiting 00:09:22 (2660): No heartbeat from core client for 30 sec - exiting 00:09:23 (2660): No heartbeat from core client for 30 sec - exiting 00:09:24 (2660): No heartbeat from core client for 30 sec - exiting 00:09:25 (2660): No heartbeat from core client for 30 sec - exiting 00:09:26 (2660): No heartbeat from core client for 30 sec - exiting 00:09:27 (2660): No heartbeat from core client for 30 sec - exiting 00:09:28 (2660): No heartbeat from core client for 30 sec - exiting 00:09:30 (2660): No heartbeat from core client for 30 sec - exiting 00:09:31 (2660): No heartbeat from core client for 30 sec - exiting 00:09:32 (2660): No heartbeat from core client for 30 sec - exiting 00:09:33 (2660): No heartbeat from core client for 30 sec - exiting 00:09:34 (2660): No heartbeat from core client for 30 sec - exiting 00:09:35 (2660): No heartbeat from core client for 30 sec - exiting 00:09:36 (2660): No heartbeat from core client for 30 sec - exiting 00:09:37 (2660): No heartbeat from core client for 30 sec - exiting 00:09:38 (2660): No heartbeat from core client for 30 sec - exiting 00:09:39 (2660): No heartbeat from core client for 30 sec - exiting 00:09:40 (2660): No heartbeat from core client for 30 sec - exiting 00:09:42 (2660): No heartbeat from core client for 30 sec - exiting 00:09:43 (2660): No heartbeat from core client for 30 sec - exiting 00:09:44 (2660): No heartbeat from core client for 30 sec - exiting 00:09:45 (2660): No heartbeat from core client for 30 sec - exiting 00:09:46 (2660): No heartbeat from core client for 30 sec - exiting 00:09:47 (2660): No heartbeat from core client for 30 sec - exiting 00:09:48 (2660): No heartbeat from core client for 30 sec - exiting 00:09:49 (2660): No heartbeat from core client for 30 sec - exiting 00:09:50 (2660): No heartbeat from core client for 30 sec - exiting 00:09:51 (2660): No heartbeat from core client for 30 sec - exiting 00:09:52 (2660): No heartbeat from core client for 30 sec - exiting 00:09:54 (2660): No heartbeat from core client for 30 sec - exiting 00:10:27 (2660): No heartbeat from core client for 30 sec - exiting 00:10:28 (2660): No heartbeat from core client for 30 sec - exiting 00:10:30 (2660): No heartbeat from core client for 30 sec - exiting 00:10:31 (2660): No heartbeat from core client for 30 sec - exiting 00:10:32 (2660): No heartbeat from core client for 30 sec - exiting 00:10:33 (2660): No heartbeat from core client for 30 sec - exiting 00:10:34 (2660): No heartbeat from core client for 30 sec - exiting 00:10:35 (2660): No heartbeat from core client for 30 sec - exiting 00:10:36 (2660): No heartbeat from core client for 30 sec - exiting 00:10:37 (2660): No heartbeat from core client for 30 sec - exiting 00:10:38 (2660): No heartbeat from core client for 30 sec - exiting 00:10:39 (2660): No heartbeat from core client for 30 sec - exiting 00:10:40 (2660): No heartbeat from core client for 30 sec - exiting 00:10:42 (2660): No heartbeat from core client for 30 sec - exiting 00:10:43 (2660): No heartbeat from core client for 30 sec - exiting 00:10:44 (2660): No heartbeat from core client for 30 sec - exiting 00:10:45 (2660): No heartbeat from core client for 30 sec - exiting 00:10:46 (2660): No heartbeat from core client for 30 sec - exiting 00:10:47 (2660): No heartbeat from core client for 30 sec - exiting 00:10:48 (2660): No heartbeat from core client for 30 sec - exiting 00:10:49 (2660): No heartbeat from core client for 30 sec - exiting 00:10:50 (2660): No heartbeat from core client for 30 sec - exiting 00:10:51 (2660): No heartbeat from core client for 30 sec - exiting 00:10:53 (2660): No heartbeat from core client for 30 sec - exiting 00:10:54 (2660): No heartbeat from core client for 30 sec - exiting 00:10:55 (2660): No heartbeat from core client for 30 sec - exiting 00:10:56 (2660): No heartbeat from core client for 30 sec - exiting 00:10:57 (2660): No heartbeat from core client for 30 sec - exiting 00:10:58 (2660): No heartbeat from core client for 30 sec - exiting 00:10:59 (2660): No heartbeat from core client for 30 sec - exiting 00:11:00 (2660): No heartbeat from core client for 30 sec - exiting 00:11:34 (2660): No heartbeat from core client for 30 sec - exiting 00:11:35 (2660): No heartbeat from core client for 30 sec - exiting 00:11:36 (2660): No heartbeat from core client for 30 sec - exiting 00:11:37 (2660): No heartbeat from core client for 30 sec - exiting 00:11:38 (2660): No heartbeat from core client for 30 sec - exiting 00:11:39 (2660): No heartbeat from core client for 30 sec - exiting 00:11:41 (2660): No heartbeat from core client for 30 sec - exiting 00:11:42 (2660): No heartbeat from core client for 30 sec - exiting 00:11:43 (2660): No heartbeat from core client for 30 sec - exiting 00:11:44 (2660): No heartbeat from core client for 30 sec - exiting 00:11:45 (2660): No heartbeat from core client for 30 sec - exiting 00:11:46 (2660): No heartbeat from core client for 30 sec - exiting 00:11:47 (2660): No heartbeat from core client for 30 sec - exiting 00:11:48 (2660): No heartbeat from core client for 30 sec - exiting 00:11:49 (2660): No heartbeat from core client for 30 sec - exiting 00:11:50 (2660): No heartbeat from core client for 30 sec - exiting 00:11:51 (2660): No heartbeat from core client for 30 sec - exiting 00:11:53 (2660): No heartbeat from core client for 30 sec - exiting 00:11:54 (2660): No heartbeat from core client for 30 sec - exiting 00:11:55 (2660): No heartbeat from core client for 30 sec - exiting 00:11:56 (2660): No heartbeat from core client for 30 sec - exiting 00:11:57 (2660): No heartbeat from core client for 30 sec - exiting 00:11:58 (2660): No heartbeat from core client for 30 sec - exiting 00:11:59 (2660): No heartbeat from core client for 30 sec - exiting 00:12:00 (2660): No heartbeat from core client for 30 sec - exiting 00:12:01 (2660): No heartbeat from core client for 30 sec - exiting 00:12:02 (2660): No heartbeat from core client for 30 sec - exiting 00:12:03 (2660): No heartbeat from core client for 30 sec - exiting 00:13:02 (2660): No heartbeat from core client for 30 sec - exiting 00:13:03 (2660): No heartbeat from core client for 30 sec - exiting 00:13:04 (2660): No heartbeat from core client for 30 sec - exiting 00:13:05 (2660): No heartbeat from core client for 30 sec - exiting 00:13:06 (2660): No heartbeat from core client for 30 sec - exiting 00:13:07 (2660): No heartbeat from core client for 30 sec - exiting 00:13:08 (2660): No heartbeat from core client for 30 sec - exiting 00:13:09 (2660): No heartbeat from core client for 30 sec - exiting 00:13:10 (2660): No heartbeat from core client for 30 sec - exiting 00:13:11 (2660): No heartbeat from core client for 30 sec - exiting 00:13:12 (2660): No heartbeat from core client for 30 sec - exiting 00:13:14 (2660): No heartbeat from core client for 30 sec - exiting 00:13:15 (2660): No heartbeat from core client for 30 sec - exiting 00:13:16 (2660): No heartbeat from core client for 30 sec - exiting 00:13:17 (2660): No heartbeat from core client for 30 sec - exiting 00:13:18 (2660): No heartbeat from core client for 30 sec - exiting 00:13:19 (2660): No heartbeat from core client for 30 sec - exiting 00:13:20 (2660): No heartbeat from core client for 30 sec - exiting 00:13:21 (2660): No heartbeat from core client for 30 sec - exiting 00:13:22 (2660): No heartbeat from core client for 30 sec - exiting 00:13:23 (2660): No heartbeat from core client for 30 sec - exiting 00:13:24 (2660): No heartbeat from core client for 30 sec - exiting 00:13:26 (2660): No heartbeat from core client for 30 sec - exiting 00:13:27 (2660): No heartbeat from core client for 30 sec - exiting 00:13:28 (2660): No heartbeat from core client for 30 sec - exiting 00:13:29 (2660): No heartbeat from core client for 30 sec - exiting 00:13:30 (2660): No heartbeat from core client for 30 sec - exiting 00:14:04 (2660): No heartbeat from core client for 30 sec - exiting 00:14:05 (2660): No heartbeat from core client for 30 sec - exiting 00:14:06 (2660): No heartbeat from core client for 30 sec - exiting 00:14:07 (2660): No heartbeat from core client for 30 sec - exiting 00:14:08 (2660): No heartbeat from core client for 30 sec - exiting 00:14:09 (2660): No heartbeat from core client for 30 sec - exiting 00:14:10 (2660): No heartbeat from core client for 30 sec - exiting 00:14:11 (2660): No heartbeat from core client for 30 sec - exiting 00:14:12 (2660): No heartbeat from core client for 30 sec - exiting 00:14:14 (2660): No heartbeat from core client for 30 sec - exiting 00:14:15 (2660): No heartbeat from core client for 30 sec - exiting 00:14:16 (2660): No heartbeat from core client for 30 sec - exiting 00:14:17 (2660): No heartbeat from core client for 30 sec - exiting 00:14:18 (2660): No heartbeat from core client for 30 sec - exiting 00:14:19 (2660): No heartbeat from core client for 30 sec - exiting 00:14:20 (2660): No heartbeat from core client for 30 sec - exiting 00:14:21 (2660): No heartbeat from core client for 30 sec - exiting 00:14:22 (2660): No heartbeat from core client for 30 sec - exiting 00:14:27 (2660): No heartbeat from core client for 30 sec - exiting 00:14:28 (2660): No heartbeat from core client for 30 sec - exiting 00:14:29 (2660): No heartbeat from core client for 30 sec - exiting 00:14:30 (2660): No heartbeat from core client for 30 sec - exiting 00:14:31 (2660): No heartbeat from core client for 30 sec - exiting 00:14:32 (2660): No heartbeat from core client for 30 sec - exiting 00:14:33 (2660): No heartbeat from core client for 30 sec - exiting 00:14:34 (2660): No heartbeat from core client for 30 sec - exiting 00:14:35 (2660): No heartbeat from core client for 30 sec - exiting 00:14:37 (2660): No heartbeat from core client for 30 sec - exiting 00:14:38 (2660): No heartbeat from core client for 30 sec - exiting 00:14:39 (2660): No heartbeat from core client for 30 sec - exiting 00:14:40 (2660): No heartbeat from core client for 30 sec - exiting 00:14:41 (2660): No heartbeat from core client for 30 sec - exiting 00:14:42 (2660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:03:20 (5144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:06:47 (2956): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3724, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3724, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3724, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3724, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3724, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3724, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish 17:09:59 (6304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Sep 2011 19:32:58	1122757	13374904	hadcm3n_t39h_1940_40_007452362_2	51,840	158,258	3.0528
14 Sep 2011 21:15:31	1122757	13374904	hadcm3n_t39h_1940_40_007452362_2	25,920	79,185	3.0550