Task 15914703

Name	hadcm3n_4ahg_2020_40_008362455_1
Workunit	8513314
Created	14 Aug 2013, 11:39:16 UTC
Sent	14 Aug 2013, 18:15:14 UTC
Report deadline	14 Nov 2013, 1:42:25 UTC
Received	7 Oct 2013, 6:37:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1254387
Run time	52 days 21 hours 49 min 38 sec
CPU time	15 hours 15 min 35 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.63 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 forrtl: The requested operation cannot be performed on a file with a user-mapped section open. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2404, iMonCtr=1 Model crash detected, will try to restart... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 forrtl: The requested operation cannot be performed on a file with a user-mapped section open. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2404, iMonCtr=1 Model crash detected, will try to restart... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 forrtl: The requested operation cannot be performed on a file with a user-mapped section open. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2404, iMonCtr=1 Model crash detected, will try to restart... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 forrtl: The requested operation cannot be performed on a file with a user-mapped section open. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2404, iMonCtr=1 Model crash detected, will try to restart... 01:58:50 (2404): No heartbeat from core client for 30 sec - exiting 01:58:51 (2404): No heartbeat from core client for 30 sec - exiting 01:58:53 (2404): No heartbeat from core client for 30 sec - exiting 01:58:54 (2404): No heartbeat from core client for 30 sec - exiting 01:58:55 (2404): No heartbeat from core client for 30 sec - exiting 01:58:56 (2404): No heartbeat from core client for 30 sec - exiting 01:58:57 (2404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... zip error: Could not create output file (was replacing the original zip file) cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_4ahg_2020_40_008362455/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Oct 2013 05:38:03	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	518,400	55,087	0.1063
02 Oct 2013 16:01:14	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	492,480	51,971	0.1055
29 Sep 2013 20:24:35	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	466,560	48,919	0.1049
27 Sep 2013 07:16:36	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	440,640	52,664	0.1195
21 Sep 2013 11:31:10	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	414,720	134,518	0.3244
20 Sep 2013 04:17:33	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	388,800	90,607	0.2330
18 Sep 2013 21:14:13	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	362,880	91,762	0.2529
17 Sep 2013 14:07:52	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	336,960	87,213	0.2588
15 Sep 2013 17:51:40	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	311,040	70,147	0.2255
13 Sep 2013 04:25:00	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	285,120	48,946	0.1717
10 Sep 2013 14:51:09	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	259,200	48,947	0.1888
08 Sep 2013 02:04:41	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	233,280	56,522	0.2423
04 Sep 2013 19:09:51	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	207,360	60,045	0.2896
02 Sep 2013 06:43:18	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	181,440	51,493	0.2838
31 Aug 2013 04:43:07	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	155,520	45,184	0.2905
27 Aug 2013 20:14:14	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	129,600	6,942	0.0536
25 Aug 2013 21:40:46	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	103,680	7,505	0.0724
21 Aug 2013 03:35:29	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	77,760	15,415	0.1982
19 Aug 2013 04:03:41	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	51,840	5,151	0.0994
16 Aug 2013 17:21:24	1254387	15914703	hadcm3n_4ahg_2020_40_008362455_1	25,920	1,800	0.0694