Task 15442691

Name	hadcm3n_zilp_1880_40_008246573_0
Workunit	8401697
Created	21 Nov 2012, 2:11:17 UTC
Sent	21 Nov 2012, 2:11:27 UTC
Report deadline	20 Feb 2013, 9:38:38 UTC
Received	16 Dec 2012, 6:22:19 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1099480
Run time	9 days 14 hours 50 min 5 sec
CPU time	8 days 8 hours 18 min 4 sec
Validate state	Invalid
Credit	4,665.60
Device peak FLOPS	2.74 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5716, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5716, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=940, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=940, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:06:58 (3088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4520, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4520, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4968, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4968, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4968, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5580, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4312, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Dec 2012 03:22:31	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	388,800	694,906	1.7873
14 Dec 2012 12:15:17	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	362,880	650,697	1.7931
14 Dec 2012 12:15:17	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	336,960	603,109	1.7899
14 Dec 2012 12:15:17	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	311,040	555,789	1.7869
14 Dec 2012 12:15:17	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	285,120	508,038	1.7818
14 Dec 2012 12:15:17	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	259,200	461,407	1.7801
14 Dec 2012 12:15:17	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	233,280	414,611	1.7773
06 Dec 2012 05:17:44	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	207,360	367,446	1.7720
04 Dec 2012 13:53:23	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	181,440	319,242	1.7595
03 Dec 2012 04:53:36	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	155,520	272,589	1.7528
02 Dec 2012 02:27:27	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	129,600	226,353	1.7466
30 Nov 2012 16:34:30	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	103,680	180,835	1.7442
30 Nov 2012 03:40:44	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	77,760	135,473	1.7422
28 Nov 2012 06:26:13	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	51,840	89,810	1.7324
26 Nov 2012 07:19:47	1099480	15442691	hadcm3n_zilp_1880_40_008246573_0	25,920	43,838	1.6913