Task 15915228

Name	hadcm3n_zlip_1920_40_008364853_1
Workunit	8515712
Created	14 Aug 2013, 11:39:45 UTC
Sent	14 Aug 2013, 18:01:45 UTC
Report deadline	14 Nov 2013, 1:28:56 UTC
Received	20 Sep 2013, 1:01:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1084078
Run time	11 days 11 hours 38 min 25 sec
CPU time	8 days 4 hours 30 min 11 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	1.92 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1148, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2040, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=884, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6840, iMonCtr=1 Model crash detected, will try to restart... 06:16:33 (5264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:16:35 (5264): No heartbeat from core client for 30 sec - exiting 06:16:36 (5264): No heartbeat from core client for 30 sec - exiting 06:16:38 (5264): No heartbeat from core client for 30 sec - exiting 06:16:39 (5264): No heartbeat from core client for 30 sec - exiting 06:16:40 (5264): No heartbeat from core client for 30 sec - exiting 06:16:41 (5264): No heartbeat from core client for 30 sec - exiting 06:16:42 (5264): No heartbeat from core client for 30 sec - exiting 06:18:03 (10724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found 15:09:39 (4464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:09:41 (4464): No heartbeat from core client for 30 sec - exiting 15:09:42 (4464): No heartbeat from core client for 30 sec - exiting 15:09:43 (4464): No heartbeat from core client for 30 sec - exiting 15:09:44 (4464): No heartbeat from core client for 30 sec - exiting 15:09:45 (4464): No heartbeat from core client for 30 sec - exiting 15:09:46 (4464): No heartbeat from core client for 30 sec - exiting 15:11:37 (8384): No heartbeat from core client for 30 sec - exiting 15:11:51 (8384): No heartbeat from core client for 30 sec - exiting 15:11:52 (8384): No heartbeat from core client for 30 sec - exiting 15:11:53 (8384): No heartbeat from core client for 30 sec - exiting 15:11:54 (8384): No heartbeat from core client for 30 sec - exiting 15:11:55 (8384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:28:10 (7500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7492, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4288, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4288, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5644, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5644, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error 13 - Permission denied Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zlip_1920_40_008364853/dataout/atmos_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4444, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zlip_1920_40_008364853/dataout/atmos_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4444, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zlip_1920_40_008364853/dataout/atmos_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4444, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zlip_1920_40_008364853/dataout/atmos_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4444, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zlip_1920_40_008364853/dataout/atmos_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4444, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zlip_1920_40_008364853/dataout/atmos_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4444, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Sep 2013 09:43:18	1084078	15915228	hadcm3n_zlip_1920_40_008364853_1	285,120	705,862	2.4757
13 Sep 2013 11:56:47	1084078	15915228	hadcm3n_zlip_1920_40_008364853_1	259,200	770,420	2.9723
12 Sep 2013 14:31:28	1084078	15915228	hadcm3n_zlip_1920_40_008364853_1	233,280	695,021	2.9793
07 Sep 2013 14:45:31	1084078	15915228	hadcm3n_zlip_1920_40_008364853_1	207,360	617,649	2.9786
04 Sep 2013 17:59:33	1084078	15915228	hadcm3n_zlip_1920_40_008364853_1	181,440	540,929	2.9813
31 Aug 2013 18:09:17	1084078	15915228	hadcm3n_zlip_1920_40_008364853_1	155,520	463,881	2.9828
27 Aug 2013 13:42:39	1084078	15915228	hadcm3n_zlip_1920_40_008364853_1	129,600	385,875	2.9774
23 Aug 2013 18:13:50	1084078	15915228	hadcm3n_zlip_1920_40_008364853_1	103,680	308,818	2.9786
22 Aug 2013 09:44:51	1084078	15915228	hadcm3n_zlip_1920_40_008364853_1	77,760	232,637	2.9917
20 Aug 2013 18:36:09	1084078	15915228	hadcm3n_zlip_1920_40_008364853_1	51,840	155,908	3.0075
17 Aug 2013 12:00:13	1084078	15915228	hadcm3n_zlip_1920_40_008364853_1	25,920	79,246	3.0573