Task 15933199

Name	hadcm3n_o1hi_2020_40_008409166_0
Workunit	8560022
Created	21 Aug 2013, 23:28:32 UTC
Sent	23 Aug 2013, 13:58:54 UTC
Report deadline	22 Nov 2013, 21:26:05 UTC
Received	7 Sep 2013, 14:32:23 UTC
Server state	Over
Outcome	Computation error
Client state	Done
Exit status	22 (0x00000016) Unknown error code
Computer ID	987999
Run time	7 days 7 hours 2 min 36 sec
CPU time	7 days 7 hours 2 min 36 sec
Validate state	Invalid
Credit	4,354.56
Device peak FLOPS	2.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>5.10.30</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:29:01 (2588): No heartbeat from core client for 30 sec - exiting 14:29:02 (2588): No heartbeat from core client for 30 sec - exiting 14:29:03 (2588): No heartbeat from core client for 30 sec - exiting 14:29:04 (2588): No heartbeat from core client for 30 sec - exiting 14:29:05 (2588): No heartbeat from core client for 30 sec - exiting 14:29:06 (2588): No heartbeat from core client for 30 sec - exiting 14:29:07 (2588): No heartbeat from core client for 30 sec - exiting 14:29:09 (2588): No heartbeat from core client for 30 sec - exiting 14:29:10 (2588): No heartbeat from core client for 30 sec - exiting 14:29:11 (2588): No heartbeat from core client for 30 sec - exiting 14:29:12 (2588): No heartbeat from core client for 30 sec - exiting 14:29:13 (2588): No heartbeat from core client for 30 sec - exiting 14:29:14 (2588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:36:45 (2680): No heartbeat from core client for 30 sec - exiting 14:36:46 (2680): No heartbeat from core client for 30 sec - exiting 14:36:47 (2680): No heartbeat from core client for 30 sec - exiting 14:36:48 (2680): No heartbeat from core client for 30 sec - exiting 14:36:49 (2680): No heartbeat from core client for 30 sec - exiting 14:36:51 (2680): No heartbeat from core client for 30 sec - exiting 14:36:52 (2680): No heartbeat from core client for 30 sec - exiting 14:36:53 (2680): No heartbeat from core client for 30 sec - exiting 14:36:54 (2680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2548, iMonCtr=1 Model crash detected, will try to restart... 09:25:23 (2548): No heartbeat from core client for 30 sec - exiting 09:25:24 (2548): No heartbeat from core client for 30 sec - exiting No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2624, selfPID=2624, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:47:13 (2396): No heartbeat from core client for 30 sec - exiting 23:47:14 (2396): No heartbeat from core client for 30 sec - exiting 23:47:15 (2396): No heartbeat from core client for 30 sec - exiting 23:47:16 (2396): No heartbeat from core client for 30 sec - exiting 23:47:17 (2396): No heartbeat from core client for 30 sec - exiting 23:47:18 (2396): No heartbeat from core client for 30 sec - exiting 23:47:19 (2396): No heartbeat from core client for 30 sec - exiting 23:47:20 (2396): No heartbeat from core client for 30 sec - exiting 23:47:21 (2396): No heartbeat from core client for 30 sec - exiting 23:47:22 (2396): No heartbeat from core client for 30 sec - exiting 23:47:23 (2396): No heartbeat from core client for 30 sec - exiting 23:47:25 (2396): No heartbeat from core client for 30 sec - exiting 23:47:26 (2396): No heartbeat from core client for 30 sec - exiting 23:47:27 (2396): No heartbeat from core client for 30 sec - exiting 23:47:28 (2396): No heartbeat from core client for 30 sec - exiting 23:47:29 (2396): No heartbeat from core client for 30 sec - exiting 23:47:30 (2396): No heartbeat from core client for 30 sec - exiting 23:47:31 (2396): No heartbeat from core client for 30 sec - exiting 23:47:32 (2396): No heartbeat from core client for 30 sec - exiting 23:47:33 (2396): No heartbeat from core client for 30 sec - exiting 23:47:34 (2396): No heartbeat from core client for 30 sec - exiting 23:47:36 (2396): No heartbeat from core client for 30 sec - exiting 23:47:37 (2396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:53:18 (2364): No heartbeat from core client for 30 sec - exiting 23:53:19 (2364): No heartbeat from core client for 30 sec - exiting 23:53:20 (2364): No heartbeat from core client for 30 sec - exiting 23:53:21 (2364): No heartbeat from core client for 30 sec - exiting 23:53:22 (2364): No heartbeat from core client for 30 sec - exiting 23:53:23 (2364): No heartbeat from core client for 30 sec - exiting 23:53:24 (2364): No heartbeat from core client for 30 sec - exiting 23:53:25 (2364): No heartbeat from core client for 30 sec - exiting 23:53:26 (2364): No heartbeat from core client for 30 sec - exiting 23:53:27 (2364): No heartbeat from core client for 30 sec - exiting 23:53:28 (2364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:50:47 (2428): No heartbeat from core client for 30 sec - exiting 04:50:48 (2428): No heartbeat from core client for 30 sec - exiting 04:50:49 (2428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:51:11 (1496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:38:04 (2504): Can't set up shared mem: -1. Will run in standalone mode. Called boinc_finish 10:52:39 (2284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Sep 2013 22:26:13	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	362,880	625,673	1.7242
02 Sep 2013 06:23:07	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	336,960	579,326	1.7193
01 Sep 2013 10:44:50	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	311,040	534,071	1.7170
31 Aug 2013 17:13:47	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	285,120	488,967	1.7150
30 Aug 2013 17:41:15	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	259,200	443,084	1.7094
30 Aug 2013 01:28:17	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	233,280	398,451	1.7080
29 Aug 2013 06:03:29	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	207,360	354,176	1.7080
28 Aug 2013 13:49:46	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	181,440	309,863	1.7078
27 Aug 2013 19:14:01	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	155,520	265,628	1.7080
27 Aug 2013 01:04:07	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	129,600	221,575	1.7097
26 Aug 2013 08:19:17	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	103,680	177,707	1.7140
25 Aug 2013 14:19:14	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	77,760	134,401	1.7284
25 Aug 2013 00:26:32	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	51,840	89,577	1.7280
24 Aug 2013 04:06:58	987999	15933199	hadcm3n_o1hi_2020_40_008409166_0	25,920	43,550	1.6802