Task 14644853

Name	hadcm3n_yj8i_1980_40_007956436_0
Workunit	8111548
Created	9 May 2012, 2:04:26 UTC
Sent	10 May 2012, 11:28:52 UTC
Report deadline	9 Aug 2012, 18:56:03 UTC
Received	17 Sep 2012, 10:14:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	812178
Run time	4 days 12 hours 13 min 33 sec
CPU time	3 days 4 hours 13 min 16 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	2.03 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:13:21 (3276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:14:57 (2808): No heartbeat from core client for 30 sec - exiting 21:14:58 (2808): No heartbeat from core client for 30 sec - exiting 21:14:59 (2808): No heartbeat from core client for 30 sec - exiting 21:15:00 (2808): No heartbeat from core client for 30 sec - exiting 21:15:01 (2808): No heartbeat from core client for 30 sec - exiting 21:15:02 (2808): No heartbeat from core client for 30 sec - exiting 21:15:03 (2808): No heartbeat from core client for 30 sec - exiting 21:15:04 (2808): No heartbeat from core client for 30 sec - exiting 21:15:05 (2808): No heartbeat from core client for 30 sec - exiting 21:15:06 (2808): No heartbeat from core client for 30 sec - exiting 21:15:07 (2808): No heartbeat from core client for 30 sec - exiting 21:15:08 (2808): No heartbeat from core client for 30 sec - exiting 21:15:09 (2808): No heartbeat from core client for 30 sec - exiting 21:15:10 (2808): No heartbeat from core client for 30 sec - exiting 21:15:11 (2808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=500, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:40:20 (2176): No heartbeat from core client for 30 sec - exiting 19:40:21 (2176): No heartbeat from core client for 30 sec - exiting 19:40:22 (2176): No heartbeat from core client for 30 sec - exiting 19:40:23 (2176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2848, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2848, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2848, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2848, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2848, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/hadcm3n_yj8i_1980_40_007956436/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2848, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Aug 2012 04:39:24	812178	14644853	hadcm3n_yj8i_1980_40_007956436_0	155,520	264,537	1.7010
16 Aug 2012 10:21:15	812178	14644853	hadcm3n_yj8i_1980_40_007956436_0	129,600	239,603	1.8488
26 Jul 2012 10:35:49	812178	14644853	hadcm3n_yj8i_1980_40_007956436_0	103,680	198,067	1.9104
09 Jul 2012 01:33:33	812178	14644853	hadcm3n_yj8i_1980_40_007956436_0	77,760	150,647	1.9373
16 May 2012 12:16:39	812178	14644853	hadcm3n_yj8i_1980_40_007956436_0	51,840	105,128	2.0279
13 May 2012 21:18:32	812178	14644853	hadcm3n_yj8i_1980_40_007956436_0	25,920	53,081	2.0479