Task 15458682

Name	hadcm3n_zjqa_1880_40_008253076_1
Workunit	8408200
Created	23 Nov 2012, 17:48:59 UTC
Sent	23 Nov 2012, 17:54:19 UTC
Report deadline	23 Feb 2013, 1:21:30 UTC
Received	4 Dec 2012, 8:56:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	735943
Run time	5 days 21 hours 5 min 29 sec
CPU time	5 days 5 hours 34 min 59 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	1.78 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:49:47 (9844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:08:00 (9124): No heartbeat from core client for 30 sec - exiting 08:08:01 (9124): No heartbeat from core client for 30 sec - exiting 08:08:02 (9124): No heartbeat from core client for 30 sec - exiting 08:08:03 (9124): No heartbeat from core client for 30 sec - exiting 08:08:04 (9124): No heartbeat from core client for 30 sec - exiting 08:08:05 (9124): No heartbeat from core client for 30 sec - exiting 08:08:06 (9124): No heartbeat from core client for 30 sec - exiting 08:08:07 (9124): No heartbeat from core client for 30 sec - exiting 08:08:08 (9124): No heartbeat from core client for 30 sec - exiting 08:08:09 (9124): No heartbeat from core client for 30 sec - exiting 08:08:10 (9124): No heartbeat from core client for 30 sec - exiting 08:08:11 (9124): No heartbeat from core client for 30 sec - exiting 08:08:12 (9124): No heartbeat from core client for 30 sec - exiting 08:08:13 (9124): No heartbeat from core client for 30 sec - exiting 08:08:14 (9124): No heartbeat from core client for 30 sec - exiting 08:08:15 (9124): No heartbeat from core client for 30 sec - exiting 08:08:16 (9124): No heartbeat from core client for 30 sec - exiting 08:08:17 (9124): No heartbeat from core client for 30 sec - exiting 08:08:18 (9124): No heartbeat from core client for 30 sec - exiting 08:08:19 (9124): No heartbeat from core client for 30 sec - exiting 08:08:20 (9124): No heartbeat from core client for 30 sec - exiting 08:08:21 (9124): No heartbeat from core client for 30 sec - exiting 08:08:22 (9124): No heartbeat from core client for 30 sec - exiting 08:08:23 (9124): No heartbeat from core client for 30 sec - exiting 08:08:24 (9124): No heartbeat from core client for 30 sec - exiting 08:08:25 (9124): No heartbeat from core client for 30 sec - exiting 08:08:26 (9124): No heartbeat from core client for 30 sec - exiting 08:08:27 (9124): No heartbeat from core client for 30 sec - exiting 08:08:28 (9124): No heartbeat from core client for 30 sec - exiting 08:08:29 (9124): No heartbeat from core client for 30 sec - exiting 08:08:30 (9124): No heartbeat from core client for 30 sec - exiting 08:08:31 (9124): No heartbeat from core client for 30 sec - exiting 08:08:32 (9124): No heartbeat from core client for 30 sec - exiting 08:08:33 (9124): No heartbeat from core client for 30 sec - exiting 08:08:34 (9124): No heartbeat from core client for 30 sec - exiting 08:08:35 (9124): No heartbeat from core client for 30 sec - exiting 08:08:36 (9124): No heartbeat from core client for 30 sec - exiting 08:08:37 (9124): No heartbeat from core client for 30 sec - exiting 08:08:38 (9124): No heartbeat from core client for 30 sec - exiting 08:08:39 (9124): No heartbeat from core client for 30 sec - exiting 08:08:40 (9124): No heartbeat from core client for 30 sec - exiting 08:08:41 (9124): No heartbeat from core client for 30 sec - exiting 08:08:42 (9124): No heartbeat from core client for 30 sec - exiting 08:08:43 (9124): No heartbeat from core client for 30 sec - exiting 08:08:44 (9124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:08:45 (9124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 08:23:10 (9412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:39:13 (3796): No heartbeat from core client for 30 sec - exiting 08:39:14 (3796): No heartbeat from core client for 30 sec - exiting 08:39:15 (3796): No heartbeat from core client for 30 sec - exiting 08:39:16 (3796): No heartbeat from core client for 30 sec - exiting 08:39:17 (3796): No heartbeat from core client for 30 sec - exiting 08:39:18 (3796): No heartbeat from core client for 30 sec - exiting 08:39:19 (3796): No heartbeat from core client for 30 sec - exiting 08:39:20 (3796): No heartbeat from core client for 30 sec - exiting 08:39:21 (3796): No heartbeat from core client for 30 sec - exiting 08:39:22 (3796): No heartbeat from core client for 30 sec - exiting 08:39:23 (3796): No heartbeat from core client for 30 sec - exiting 08:39:24 (3796): No heartbeat from core client for 30 sec - exiting 08:39:25 (3796): No heartbeat from core client for 30 sec - exiting 08:39:26 (3796): No heartbeat from core client for 30 sec - exiting 08:39:27 (3796): No heartbeat from core client for 30 sec - exiting 08:39:28 (3796): No heartbeat from core client for 30 sec - exiting 08:39:29 (3796): No heartbeat from core client for 30 sec - exiting 08:39:30 (3796): No heartbeat from core client for 30 sec - exiting 08:39:31 (3796): No heartbeat from core client for 30 sec - exiting 08:39:32 (3796): No heartbeat from core client for 30 sec - exiting 08:39:33 (3796): No heartbeat from core client for 30 sec - exiting 08:39:34 (3796): No heartbeat from core client for 30 sec - exiting 08:39:35 (3796): No heartbeat from core client for 30 sec - exiting 08:39:36 (3796): No heartbeat from core client for 30 sec - exiting 08:39:37 (3796): No heartbeat from core client for 30 sec - exiting 08:39:38 (3796): No heartbeat from core client for 30 sec - exiting 08:39:39 (3796): No heartbeat from core client for 30 sec - exiting 08:39:40 (3796): No heartbeat from core client for 30 sec - exiting 08:39:41 (3796): No heartbeat from core client for 30 sec - exiting 08:39:42 (3796): No heartbeat from core client for 30 sec - exiting 08:39:43 (3796): No heartbeat from core client for 30 sec - exiting 08:39:44 (3796): No heartbeat from core client for 30 sec - exiting 08:39:45 (3796): No heartbeat from core client for 30 sec - exiting 08:39:46 (3796): No heartbeat from core client for 30 sec - exiting 08:39:47 (3796): No heartbeat from core client for 30 sec - exiting 08:39:48 (3796): No heartbeat from core client for 30 sec - exiting 08:39:49 (3796): No heartbeat from core client for 30 sec - exiting 08:39:50 (3796): No heartbeat from core client for 30 sec - exiting 08:39:51 (3796): No heartbeat from core client for 30 sec - exiting 08:39:52 (3796): No heartbeat from core client for 30 sec - exiting 08:39:53 (3796): No heartbeat from core client for 30 sec - exiting 08:39:54 (3796): No heartbeat from core client for 30 sec - exiting 08:39:55 (3796): No heartbeat from core client for 30 sec - exiting 08:39:56 (3796): No heartbeat from core client for 30 sec - exiting 08:39:57 (3796): No heartbeat from core client for 30 sec - exiting 08:39:58 (3796): No heartbeat from core client for 30 sec - exiting 08:39:59 (3796): No heartbeat from core client for 30 sec - exiting 08:40:00 (3796): No heartbeat from core client for 30 sec - exiting 08:40:01 (3796): No heartbeat from core client for 30 sec - exiting 08:40:02 (3796): No heartbeat from core client for 30 sec - exiting 08:40:03 (3796): No heartbeat from core client for 30 sec - exiting 08:40:04 (3796): No heartbeat from core client for 30 sec - exiting 08:40:05 (3796): No heartbeat from core client for 30 sec - exiting 08:40:06 (3796): No heartbeat from core client for 30 sec - exiting 08:40:07 (3796): No heartbeat from core client for 30 sec - exiting 08:40:08 (3796): No heartbeat from core client for 30 sec - exiting 08:40:09 (3796): No heartbeat from core client for 30 sec - exiting 08:40:10 (3796): No heartbeat from core client for 30 sec - exiting 08:40:11 (3796): No heartbeat from core client for 30 sec - exiting 08:40:12 (3796): No heartbeat from core client for 30 sec - exiting 08:40:13 (3796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:14 (3796): No heartbeat from core client for 30 sec - exiting 08:40:15 (3796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:02:52 (9704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:16:48 (9816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:24:54 (9004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:25:20 (9064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:50:19 (10044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:57:30 (6072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:37:12 (3108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:37:50 (4332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:37:51 (4332): No heartbeat from core client for 30 sec - exiting 21:37:52 (4332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Dec 2012 16:30:06	735943	15458682	hadcm3n_zjqa_1880_40_008253076_1	155,520	411,062	2.6431
01 Dec 2012 11:56:53	735943	15458682	hadcm3n_zjqa_1880_40_008253076_1	129,600	338,670	2.6132
30 Nov 2012 10:47:20	735943	15458682	hadcm3n_zjqa_1880_40_008253076_1	103,680	270,710	2.6110
28 Nov 2012 07:06:20	735943	15458682	hadcm3n_zjqa_1880_40_008253076_1	77,760	204,125	2.6251
27 Nov 2012 00:16:52	735943	15458682	hadcm3n_zjqa_1880_40_008253076_1	51,840	135,964	2.6228
26 Nov 2012 03:39:02	735943	15458682	hadcm3n_zjqa_1880_40_008253076_1	25,920	68,486	2.6422