Task 15919774

Name	hadcm3n_n0bk_1960_40_008378456_1
Workunit	8529315
Created	14 Aug 2013, 17:40:51 UTC
Sent	14 Aug 2013, 17:41:27 UTC
Report deadline	14 Nov 2013, 1:08:38 UTC
Received	29 Aug 2013, 8:33:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1186467
Run time	3 days 18 hours 28 min 50 sec
CPU time	3 days 9 hours 57 min 21 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	2.89 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.5</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:17:41 (7600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:20:04 (7056): No heartbeat from core client for 30 sec - exiting 14:20:05 (7056): No heartbeat from core client for 30 sec - exiting 14:20:06 (7056): No heartbeat from core client for 30 sec - exiting 14:20:07 (7056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:21:59 (7892): No heartbeat from core client for 30 sec - exiting 19:22:00 (7892): No heartbeat from core client for 30 sec - exiting 19:22:01 (7892): No heartbeat from core client for 30 sec - exiting 19:22:02 (7892): No heartbeat from core client for 30 sec - exiting 19:22:03 (7892): No heartbeat from core client for 30 sec - exiting 19:22:04 (7892): No heartbeat from core client for 30 sec - exiting 19:22:05 (7892): No heartbeat from core client for 30 sec - exiting 19:22:06 (7892): No heartbeat from core client for 30 sec - exiting 19:22:07 (7892): No heartbeat from core client for 30 sec - exiting 19:22:08 (7892): No heartbeat from core client for 30 sec - exiting 19:22:09 (7892): No heartbeat from core client for 30 sec - exiting 19:22:10 (7892): No heartbeat from core client for 30 sec - exiting 19:22:11 (7892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:33:44 (7036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:03:59 (7524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9272, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:20:51 (4104): No heartbeat from core client for 30 sec - exiting 15:20:52 (4104): No heartbeat from core client for 30 sec - exiting 15:20:53 (4104): No heartbeat from core client for 30 sec - exiting 15:20:54 (4104): No heartbeat from core client for 30 sec - exiting 15:20:55 (4104): No heartbeat from core client for 30 sec - exiting 15:20:56 (4104): No heartbeat from core client for 30 sec - exiting 15:20:57 (4104): No heartbeat from core client for 30 sec - exiting 15:20:58 (4104): No heartbeat from core client for 30 sec - exiting 15:20:59 (4104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:04:44 (9652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:04:45 (9652): No heartbeat from core client for 30 sec - exiting 22:06:31 (11776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:06:32 (11776): No heartbeat from core client for 30 sec - exiting 22:06:33 (11776): No heartbeat from core client for 30 sec - exiting 22:06:34 (11776): No heartbeat from core client for 30 sec - exiting 22:06:35 (11776): No heartbeat from core client for 30 sec - exiting 22:06:36 (11776): No heartbeat from core client for 30 sec - exiting 22:06:37 (11776): No heartbeat from core client for 30 sec - exiting 22:06:38 (11776): No heartbeat from core client for 30 sec - exiting 22:06:39 (11776): No heartbeat from core client for 30 sec - exiting 22:06:40 (11776): No heartbeat from core client for 30 sec - exiting 22:08:48 (4992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:41 (11564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:42 (11564): No heartbeat from core client for 30 sec - exiting 22:10:43 (11564): No heartbeat from core client for 30 sec - exiting 22:10:44 (11564): No heartbeat from core client for 30 sec - exiting 22:10:45 (11564): No heartbeat from core client for 30 sec - exiting 22:10:46 (11564): No heartbeat from core client for 30 sec - exiting 22:10:47 (11564): No heartbeat from core client for 30 sec - exiting 22:10:48 (11564): No heartbeat from core client for 30 sec - exiting 22:36:52 (296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:59:14 (1516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:44:46 (5308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 05:56:13 (13552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:54:59 (22620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Aug 2013 21:41:52	1186467	15919774	hadcm3n_n0bk_1960_40_008378456_1	155,520	294,963	1.8966
25 Aug 2013 16:39:43	1186467	15919774	hadcm3n_n0bk_1960_40_008378456_1	129,600	248,297	1.9159
24 Aug 2013 01:06:07	1186467	15919774	hadcm3n_n0bk_1960_40_008378456_1	103,680	201,645	1.9449
23 Aug 2013 11:08:35	1186467	15919774	hadcm3n_n0bk_1960_40_008378456_1	77,760	149,503	1.9226
17 Aug 2013 01:23:23	1186467	15919774	hadcm3n_n0bk_1960_40_008378456_1	51,840	98,028	1.8910
15 Aug 2013 22:36:54	1186467	15919774	hadcm3n_n0bk_1960_40_008378456_1	25,920	49,037	1.8919