Task 16059232

Name	hadcm3n_4n0l_1980_40_008322719_4
Workunit	8473854
Created	6 Oct 2013, 21:01:37 UTC
Sent	6 Oct 2013, 21:26:39 UTC
Report deadline	6 Jan 2014, 4:53:50 UTC
Received	15 Nov 2013, 21:14:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1227307
Run time	4 days 17 hours 17 min 39 sec
CPU time	4 days 7 hours 35 min 2 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	3.42 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4260, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5608, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5264, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4828, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6068, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5932, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2884, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=336, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6368, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5676, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3916, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4324, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6132, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5804, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7020, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4n0lko.pjj5c10 Error converting file to netcdf: dataout/4n0lko.pij5c10 Error converting file to netcdf: dataout/4n0lko.pfj5c10 Error converting file to netcdf: dataout/4n0lka.phj5c10 Error converting file to netcdf: dataout/4n0lka.pgj5c10 Error converting file to netcdf: dataout/4n0lka.pej5c10 Error converting file to netcdf: dataout/4n0lka.pdj5c10 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=932, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4636, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Nov 2013 19:47:55	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	414,720	364,879	0.8798
12 Nov 2013 16:20:55	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	388,800	342,071	0.8798
09 Nov 2013 22:25:27	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	362,880	319,312	0.8799
07 Nov 2013 21:54:07	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	336,960	296,453	0.8798
03 Nov 2013 20:04:38	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	311,040	273,655	0.8798
02 Nov 2013 14:53:35	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	285,120	250,595	0.8789
01 Nov 2013 21:33:42	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	259,200	227,391	0.8773
30 Oct 2013 21:54:32	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	233,280	204,500	0.8766
27 Oct 2013 20:18:13	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	207,360	182,602	0.8806
26 Oct 2013 08:16:56	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	181,440	159,825	0.8809
23 Oct 2013 20:36:47	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	155,520	137,070	0.8814
19 Oct 2013 17:02:03	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	129,600	114,269	0.8817
14 Oct 2013 20:58:34	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	103,680	91,383	0.8814
13 Oct 2013 08:39:13	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	77,760	68,617	0.8824
12 Oct 2013 07:24:16	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	51,840	45,503	0.8778
08 Oct 2013 21:52:26	1227307	16059232	hadcm3n_4n0l_1980_40_008322719_4	25,920	22,674	0.8748