Task 13749458

Name	hadcm3n_y7qn_1940_40_007614505_1
Workunit	7792635
Created	8 Dec 2011, 1:56:40 UTC
Sent	8 Dec 2011, 12:40:59 UTC
Report deadline	8 Mar 2012, 20:08:10 UTC
Received	9 Jan 2012, 21:54:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1177788
Run time	13 days 10 hours 7 min 23 sec
CPU time	13 days 5 hours 8 min 30 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.25 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:54:18 (2384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2072, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:37:10 (5492): No heartbeat from core client for 30 sec - exiting 19:37:11 (5492): No heartbeat from core client for 30 sec - exiting 19:37:12 (5492): No heartbeat from core client for 30 sec - exiting 19:37:13 (5492): No heartbeat from core client for 30 sec - exiting 19:37:14 (5492): No heartbeat from core client for 30 sec - exiting 19:37:15 (5492): No heartbeat from core client for 30 sec - exiting 19:37:16 (5492): No heartbeat from core client for 30 sec - exiting 19:37:17 (5492): No heartbeat from core client for 30 sec - exiting 19:37:18 (5492): No heartbeat from core client for 30 sec - exiting 19:37:19 (5492): No heartbeat from core client for 30 sec - exiting 19:37:20 (5492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:06:24 (6084): No heartbeat from core client for 30 sec - exiting 10:06:25 (6084): No heartbeat from core client for 30 sec - exiting 10:06:26 (6084): No heartbeat from core client for 30 sec - exiting 10:06:27 (6084): No heartbeat from core client for 30 sec - exiting 10:06:28 (6084): No heartbeat from core client for 30 sec - exiting 10:06:29 (6084): No heartbeat from core client for 30 sec - exiting 10:06:30 (6084): No heartbeat from core client for 30 sec - exiting 10:06:31 (6084): No heartbeat from core client for 30 sec - exiting 10:06:32 (6084): No heartbeat from core client for 30 sec - exiting 10:06:33 (6084): No heartbeat from core client for 30 sec - exiting 10:06:34 (6084): No heartbeat from core client for 30 sec - exiting 10:06:35 (6084): No heartbeat from core client for 30 sec - exiting 10:06:36 (6084): No heartbeat from core client for 30 sec - exiting 10:06:37 (6084): No heartbeat from core client for 30 sec - exiting 10:06:38 (6084): No heartbeat from core client for 30 sec - exiting 10:06:39 (6084): No heartbeat from core client for 30 sec - exiting 10:06:40 (6084): No heartbeat from core client for 30 sec - exiting 10:06:41 (6084): No heartbeat from core client for 30 sec - exiting 10:06:42 (6084): No heartbeat from core client for 30 sec - exiting 10:06:43 (6084): No heartbeat from core client for 30 sec - exiting 10:06:44 (6084): No heartbeat from core client for 30 sec - exiting 10:06:45 (6084): No heartbeat from core client for 30 sec - exiting 10:06:46 (6084): No heartbeat from core client for 30 sec - exiting 10:06:47 (6084): No heartbeat from core client for 30 sec - exiting 10:06:48 (6084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/y7qnko.pjf9c10 Error converting file to netcdf: dataout/y7qnko.pif9c10 Error converting file to netcdf: dataout/y7qnko.pff9c10 Error converting file to netcdf: dataout/y7qnka.phf9c10 Error converting file to netcdf: dataout/y7qnka.pgf9c10 Error converting file to netcdf: dataout/y7qnka.pef9c10 Error converting file to netcdf: dataout/y7qnka.pdf9c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5360, selfPID=5360, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/y7qnko.pjg0c10 Error converting file to netcdf: dataout/y7qnko.pig0c10 Error converting file to netcdf: dataout/y7qnko.pfg0c10 CPDN Monitor - Quit request from BOINC... zip error: Could not create output file (was replacing the original zip file) cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y7qn_1940_40_007614505/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Jan 2012 20:55:06	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	518,400	1,141,825	2.2026
02 Jan 2012 00:02:46	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	492,480	1,083,647	2.2004
27 Dec 2011 02:31:01	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	466,560	1,026,692	2.2006
21 Dec 2011 00:40:48	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	440,640	968,903	2.1989
20 Dec 2011 10:44:49	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	414,720	911,397	2.1976
19 Dec 2011 15:49:21	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	388,800	854,043	2.1966
18 Dec 2011 22:35:13	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	362,880	796,205	2.1941
18 Dec 2011 10:02:45	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	336,960	738,653	2.1921
17 Dec 2011 11:04:24	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	311,040	680,883	2.1891
16 Dec 2011 18:57:28	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	285,120	623,514	2.1868
16 Dec 2011 01:59:47	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	259,200	566,398	2.1852
14 Dec 2011 23:31:43	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	233,280	508,925	2.1816
14 Dec 2011 10:43:19	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	207,360	451,895	2.1793
13 Dec 2011 13:37:47	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	181,440	396,109	2.1831
12 Dec 2011 18:07:00	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	155,520	340,066	2.1866
12 Dec 2011 01:01:18	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	129,600	283,648	2.1886
11 Dec 2011 10:42:14	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	103,680	228,153	2.2005
10 Dec 2011 13:57:12	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	77,760	170,941	2.1983
09 Dec 2011 21:58:18	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	51,840	113,892	2.1970
09 Dec 2011 10:28:03	1177788	13749458	hadcm3n_y7qn_1940_40_007614505_1	25,920	57,182	2.2061