Task 13335746

Name	hadcm3n_o2kr_1900_40_007439454_0
Workunit	7636957
Created	5 Sep 2011, 18:16:38 UTC
Sent	6 Sep 2011, 8:26:02 UTC
Report deadline	6 Dec 2011, 15:53:13 UTC
Received	20 Sep 2011, 10:02:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1087057
Run time	1 days 1 hours 45 min 54 sec
CPU time	1 days 1 hours 45 min 54 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	2.39 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.2.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 17:03:10 (4920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:12:02 (8156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:12:03 (8156): No heartbeat from core client for 30 sec - exiting 17:12:04 (8156): No heartbeat from core client for 30 sec - exiting 17:12:05 (8156): No heartbeat from core client for 30 sec - exiting 17:12:06 (8156): No heartbeat from core client for 30 sec - exiting 17:12:07 (8156): No heartbeat from core client for 30 sec - exiting 17:28:36 (7144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:51 (2424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:52 (2424): No heartbeat from core client for 30 sec - exiting 17:31:53 (2424): No heartbeat from core client for 30 sec - exiting 17:31:54 (2424): No heartbeat from core client for 30 sec - exiting 17:31:55 (2424): No heartbeat from core client for 30 sec - exiting 17:31:56 (2424): No heartbeat from core client for 30 sec - exiting 17:31:57 (2424): No heartbeat from core client for 30 sec - exiting 17:31:58 (2424): No heartbeat from core client for 30 sec - exiting 17:31:59 (2424): No heartbeat from core client for 30 sec - exiting 17:32:00 (2424): No heartbeat from core client for 30 sec - exiting 17:32:01 (2424): No heartbeat from core client for 30 sec - exiting 17:38:21 (6792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:39:45 (6792): No heartbeat from core client for 30 sec - exiting 17:39:46 (6792): No heartbeat from core client for 30 sec - exiting 17:39:47 (6792): No heartbeat from core client for 30 sec - exiting 17:39:48 (6792): No heartbeat from core client for 30 sec - exiting 17:39:49 (6792): No heartbeat from core client for 30 sec - exiting 17:39:50 (6792): No heartbeat from core client for 30 sec - exiting 17:39:51 (6792): No heartbeat from core client for 30 sec - exiting 17:39:52 (6792): No heartbeat from core client for 30 sec - exiting 17:39:53 (6792): No heartbeat from core client for 30 sec - exiting 17:39:54 (6792): No heartbeat from core client for 30 sec - exiting 17:45:03 (6852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:02 (672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:49 (4680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:12:41 (4680): No heartbeat from core client for 30 sec - exiting 18:12:42 (4680): No heartbeat from core client for 30 sec - exiting 18:12:43 (4680): No heartbeat from core client for 30 sec - exiting 18:12:44 (4680): No heartbeat from core client for 30 sec - exiting 18:12:45 (4680): No heartbeat from core client for 30 sec - exiting 18:12:46 (4680): No heartbeat from core client for 30 sec - exiting 18:12:47 (4680): No heartbeat from core client for 30 sec - exiting 18:12:48 (4680): No heartbeat from core client for 30 sec - exiting 18:12:49 (4680): No heartbeat from core client for 30 sec - exiting 18:12:50 (4680): No heartbeat from core client for 30 sec - exiting 18:16:01 (7816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:29 (6584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:30 (6584): No heartbeat from core client for 30 sec - exiting 18:20:31 (6584): No heartbeat from core client for 30 sec - exiting 18:20:32 (6584): No heartbeat from core client for 30 sec - exiting 18:20:33 (6584): No heartbeat from core client for 30 sec - exiting 18:20:34 (6584): No heartbeat from core client for 30 sec - exiting 18:20:35 (6584): No heartbeat from core client for 30 sec - exiting 18:20:36 (6584): No heartbeat from core client for 30 sec - exiting 18:20:37 (6584): No heartbeat from core client for 30 sec - exiting 18:20:38 (6584): No heartbeat from core client for 30 sec - exiting 18:20:39 (6584): No heartbeat from core client for 30 sec - exiting 18:21:36 (920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:22:38 (3332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:28:52 (7324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:28:53 (7324): No heartbeat from core client for 30 sec - exiting 18:29:30 (5476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:51 (2228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:52 (2228): No heartbeat from core client for 30 sec - exiting 18:34:53 (2228): No heartbeat from core client for 30 sec - exiting 18:34:54 (2228): No heartbeat from core client for 30 sec - exiting 18:34:55 (2228): No heartbeat from core client for 30 sec - exiting 18:34:56 (2228): No heartbeat from core client for 30 sec - exiting 18:34:57 (2228): No heartbeat from core client for 30 sec - exiting 18:34:58 (2228): No heartbeat from core client for 30 sec - exiting 18:34:59 (2228): No heartbeat from core client for 30 sec - exiting 18:35:00 (2228): No heartbeat from core client for 30 sec - exiting 18:35:01 (2228): No heartbeat from core client for 30 sec - exiting 18:36:53 (7564): No heartbeat from core client for 30 sec - exiting 18:36:54 (7564): No heartbeat from core client for 30 sec - exiting 18:36:55 (7564): No heartbeat from core client for 30 sec - exiting 18:36:56 (7564): No heartbeat from core client for 30 sec - exiting 18:36:57 (7564): No heartbeat from core client for 30 sec - exiting 18:36:58 (7564): No heartbeat from core client for 30 sec - exiting 18:36:59 (7564): No heartbeat from core client for 30 sec - exiting 18:37:00 (7564): No heartbeat from core client for 30 sec - exiting 18:37:01 (7564): No heartbeat from core client for 30 sec - exiting 18:37:02 (7564): No heartbeat from core client for 30 sec - exiting 18:37:04 (7564): No heartbeat from core client for 30 sec - exiting 18:37:05 (7564): No heartbeat from core client for 30 sec - exiting 18:37:06 (7564): No heartbeat from core client for 30 sec - exiting 18:37:07 (7564): No heartbeat from core client for 30 sec - exiting 18:37:08 (7564): No heartbeat from core client for 30 sec - exiting 18:37:09 (7564): No heartbeat from core client for 30 sec - exiting 18:37:10 (7564): No heartbeat from core client for 30 sec - exiting 18:37:11 (7564): No heartbeat from core client for 30 sec - exiting 18:37:12 (7564): No heartbeat from core client for 30 sec - exiting 18:37:13 (7564): No heartbeat from core client for 30 sec - exiting 18:37:14 (7564): No heartbeat from core client for 30 sec - exiting 18:37:15 (7564): No heartbeat from core client for 30 sec - exiting 18:37:16 (7564): No heartbeat from core client for 30 sec - exiting 18:37:17 (7564): No heartbeat from core client for 30 sec - exiting 18:37:18 (7564): No heartbeat from core client for 30 sec - exiting 18:37:19 (7564): No heartbeat from core client for 30 sec - exiting 18:37:20 (7564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:23 (6208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:24 (6208): No heartbeat from core client for 30 sec - exiting 18:38:25 (6208): No heartbeat from core client for 30 sec - exiting 18:38:26 (6208): No heartbeat from core client for 30 sec - exiting 18:38:27 (6208): No heartbeat from core client for 30 sec - exiting 18:38:28 (6208): No heartbeat from core client for 30 sec - exiting 18:38:29 (6208): No heartbeat from core client for 30 sec - exiting 18:38:30 (6208): No heartbeat from core client for 30 sec - exiting 18:38:31 (6208): No heartbeat from core client for 30 sec - exiting 18:38:32 (6208): No heartbeat from core client for 30 sec - exiting 18:38:33 (6208): No heartbeat from core client for 30 sec - exiting 18:43:18 (2000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:59:59 (7164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:00:00 (7164): No heartbeat from core client for 30 sec - exiting 19:00:01 (7164): No heartbeat from core client for 30 sec - exiting 19:00:02 (7164): No heartbeat from core client for 30 sec - exiting 19:00:03 (7164): No heartbeat from core client for 30 sec - exiting 19:00:04 (7164): No heartbeat from core client for 30 sec - exiting 19:00:05 (7164): No heartbeat from core client for 30 sec - exiting 19:00:06 (7164): No heartbeat from core client for 30 sec - exiting 19:00:07 (7164): No heartbeat from core client for 30 sec - exiting 19:00:08 (7164): No heartbeat from core client for 30 sec - exiting 19:00:09 (7164): No heartbeat from core client for 30 sec - exiting 19:08:42 (7680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:11:00 (7832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5556, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7156, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7156, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7156, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7156, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7156, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7156, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Sep 2011 14:37:14	1087057	13335746	hadcm3n_o2kr_1900_40_007439454_0	155,520	116,296	0.7478
16 Sep 2011 10:20:36	1087057	13335746	hadcm3n_o2kr_1900_40_007439454_0	129,600	135,308	1.0440
15 Sep 2011 09:16:41	1087057	13335746	hadcm3n_o2kr_1900_40_007439454_0	103,680	85,703	0.8266
13 Sep 2011 14:36:43	1087057	13335746	hadcm3n_o2kr_1900_40_007439454_0	77,760	84,025	1.0806
12 Sep 2011 13:24:03	1087057	13335746	hadcm3n_o2kr_1900_40_007439454_0	51,840	91,649	1.7679
07 Sep 2011 15:10:09	1087057	13335746	hadcm3n_o2kr_1900_40_007439454_0	25,920	48,364	1.8659