Task 13654278

Name	hadcm3n_o42f_1980_40_007538384_4
Workunit	7735616
Created	22 Nov 2011, 19:52:53 UTC
Sent	22 Nov 2011, 19:56:54 UTC
Report deadline	22 Feb 2012, 3:24:05 UTC
Received	25 Dec 2011, 23:07:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1181023
Run time	6 days 16 hours 42 min 35 sec
CPU time	6 days 6 hours 58 min 23 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.95 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3160, iMonCtr=1 Model crash detected, will try to restart... 11:22:23 (3300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=428, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2720, iMonCtr=1 Model crash detected, will try to restart... 19:08:22 (6068): No heartbeat from core client for 30 sec - exiting 19:08:23 (6068): No heartbeat from core client for 30 sec - exiting 19:08:24 (6068): No heartbeat from core client for 30 sec - exiting 19:08:26 (6068): No heartbeat from core client for 30 sec - exiting 19:08:27 (6068): No heartbeat from core client for 30 sec - exiting 19:08:28 (6068): No heartbeat from core client for 30 sec - exiting 19:08:29 (6068): No heartbeat from core client for 30 sec - exiting 19:08:30 (6068): No heartbeat from core client for 30 sec - exiting 19:08:31 (6068): No heartbeat from core client for 30 sec - exiting 19:08:32 (6068): No heartbeat from core client for 30 sec - exiting 19:08:33 (6068): No heartbeat from core client for 30 sec - exiting 19:08:34 (6068): No heartbeat from core client for 30 sec - exiting 19:08:35 (6068): No heartbeat from core client for 30 sec - exiting 19:08:36 (6068): No heartbeat from core client for 30 sec - exiting 19:08:38 (6068): No heartbeat from core client for 30 sec - exiting 19:08:39 (6068): No heartbeat from core client for 30 sec - exiting 19:08:40 (6068): No heartbeat from core client for 30 sec - exiting 19:08:41 (6068): No heartbeat from core client for 30 sec - exiting 19:08:42 (6068): No heartbeat from core client for 30 sec - exiting 19:08:43 (6068): No heartbeat from core client for 30 sec - exiting 19:08:44 (6068): No heartbeat from core client for 30 sec - exiting 19:08:45 (6068): No heartbeat from core client for 30 sec - exiting 19:08:46 (6068): No heartbeat from core client for 30 sec - exiting 19:08:47 (6068): No heartbeat from core client for 30 sec - exiting 19:08:48 (6068): No heartbeat from core client for 30 sec - exiting 19:08:50 (6068): No heartbeat from core client for 30 sec - exiting 19:08:51 (6068): No heartbeat from core client for 30 sec - exiting 19:08:52 (6068): No heartbeat from core client for 30 sec - exiting 19:08:53 (6068): No heartbeat from core client for 30 sec - exiting 19:08:54 (6068): No heartbeat from core client for 30 sec - exiting 19:08:55 (6068): No heartbeat from core client for 30 sec - exiting 19:08:56 (6068): No heartbeat from core client for 30 sec - exiting 19:08:57 (6068): No heartbeat from core client for 30 sec - exiting 19:08:58 (6068): No heartbeat from core client for 30 sec - exiting 19:08:59 (6068): No heartbeat from core client for 30 sec - exiting 19:09:00 (6068): No heartbeat from core client for 30 sec - exiting 19:09:02 (6068): No heartbeat from core client for 30 sec - exiting 19:09:03 (6068): No heartbeat from core client for 30 sec - exiting 19:09:04 (6068): No heartbeat from core client for 30 sec - exiting 19:09:05 (6068): No heartbeat from core client for 30 sec - exiting 19:09:06 (6068): No heartbeat from core client for 30 sec - exiting 19:09:07 (6068): No heartbeat from core client for 30 sec - exiting 19:09:08 (6068): No heartbeat from core client for 30 sec - exiting 19:09:09 (6068): No heartbeat from core client for 30 sec - exiting 19:09:10 (6068): No heartbeat from core client for 30 sec - exiting 19:09:11 (6068): No heartbeat from core client for 30 sec - exiting 19:09:13 (6068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5904, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2824, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2100, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2100, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7852, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3768, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2352, iMonCtr=1 Model crash detected, will try to restart... 13:07:28 (5248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:19:29 (2940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3056, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2648, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2648, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2648, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2504, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1088, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=992, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3048, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2152, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2152, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2152, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2152, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2152, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2152, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Dec 2011 14:20:02	1181023	13654278	hadcm3n_o42f_1980_40_007538384_4	259,200	528,140	2.0376
23 Dec 2011 23:33:08	1181023	13654278	hadcm3n_o42f_1980_40_007538384_4	233,280	479,634	2.0560
22 Dec 2011 18:04:57	1181023	13654278	hadcm3n_o42f_1980_40_007538384_4	207,360	433,018	2.0882
20 Dec 2011 17:20:39	1181023	13654278	hadcm3n_o42f_1980_40_007538384_4	181,440	383,355	2.1128
18 Dec 2011 22:15:07	1181023	13654278	hadcm3n_o42f_1980_40_007538384_4	155,520	335,895	2.1598
16 Dec 2011 17:36:54	1181023	13654278	hadcm3n_o42f_1980_40_007538384_4	129,600	278,679	2.1503
12 Dec 2011 20:44:56	1181023	13654278	hadcm3n_o42f_1980_40_007538384_4	103,680	220,480	2.1265
10 Dec 2011 16:22:48	1181023	13654278	hadcm3n_o42f_1980_40_007538384_4	77,760	165,610	2.1298
08 Dec 2011 15:59:57	1181023	13654278	hadcm3n_o42f_1980_40_007538384_4	51,840	113,654	2.1924
01 Dec 2011 21:23:50	1181023	13654278	hadcm3n_o42f_1980_40_007538384_4	25,920	61,485	2.3721