Task 15684158

Name	hadcm3n_u2nt_2020_40_008336019_0
Workunit	8486880
Created	26 Mar 2013, 19:23:00 UTC
Sent	26 Mar 2013, 19:23:27 UTC
Report deadline	26 Jun 2013, 2:50:38 UTC
Received	5 Nov 2013, 16:54:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	967258
Run time	9 days 13 hours 28 min 38 sec
CPU time	7 days 20 hours 26 min 53 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	1.86 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6904, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5236, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:15:00 (6408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6748, iMonCtr=1 Model crash detected, will try to restart... 17:42:32 (5340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:11:55 (1136): No heartbeat from core client for 30 sec - exiting 21:11:56 (1136): No heartbeat from core client for 30 sec - exiting 21:11:57 (1136): No heartbeat from core client for 30 sec - exiting 21:11:59 (1136): No heartbeat from core client for 30 sec - exiting 21:12:00 (1136): No heartbeat from core client for 30 sec - exiting 21:12:01 (1136): No heartbeat from core client for 30 sec - exiting 21:12:02 (1136): No heartbeat from core client for 30 sec - exiting 21:12:03 (1136): No heartbeat from core client for 30 sec - exiting 21:12:04 (1136): No heartbeat from core client for 30 sec - exiting 21:12:05 (1136): No heartbeat from core client for 30 sec - exiting 21:12:07 (1136): No heartbeat from core client for 30 sec - exiting 21:12:08 (1136): No heartbeat from core client for 30 sec - exiting 21:12:09 (1136): No heartbeat from core client for 30 sec - exiting 21:12:10 (1136): No heartbeat from core client for 30 sec - exiting 21:12:11 (1136): No heartbeat from core client for 30 sec - exiting 21:12:12 (1136): No heartbeat from core client for 30 sec - exiting 21:12:13 (1136): No heartbeat from core client for 30 sec - exiting 21:12:14 (1136): No heartbeat from core client for 30 sec - exiting 21:12:16 (1136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4904, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:21:30 (6592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:21:58 (3108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7664, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7048, iMonCtr=1 Model crash detected, will try to restart... 13:26:26 (7560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:20:24 (2524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:13:37 (5744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7196, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7196, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7568, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5664, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 18:39:53 (5336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:39:54 (5336): No heartbeat from core client for 30 sec - exiting 18:39:55 (5336): No heartbeat from core client for 30 sec - exiting 18:39:56 (5336): No heartbeat from core client for 30 sec - exiting 18:39:57 (5336): No heartbeat from core client for 30 sec - exiting 18:39:58 (5336): No heartbeat from core client for 30 sec - exiting 18:39:59 (5336): No heartbeat from core client for 30 sec - exiting 18:40:00 (5336): No heartbeat from core client for 30 sec - exiting 18:43:40 (1004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:41 (1004): No heartbeat from core client for 30 sec - exiting 18:43:42 (1004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 20:33:37 (6396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5324, iMonCtr=1 Model crash detected, will try to restart... 19:20:34 (7052): No heartbeat from core client for 30 sec - exiting 19:20:35 (7052): No heartbeat from core client for 30 sec - exiting 19:20:37 (7052): No heartbeat from core client for 30 sec - exiting 19:20:38 (7052): No heartbeat from core client for 30 sec - exiting 19:20:39 (7052): No heartbeat from core client for 30 sec - exiting 19:20:40 (7052): No heartbeat from core client for 30 sec - exiting 19:20:41 (7052): No heartbeat from core client for 30 sec - exiting 19:20:42 (7052): No heartbeat from core client for 30 sec - exiting 19:20:43 (7052): No heartbeat from core client for 30 sec - exiting 19:20:44 (7052): No heartbeat from core client for 30 sec - exiting 19:20:45 (7052): No heartbeat from core client for 30 sec - exiting 19:20:46 (7052): No heartbeat from core client for 30 sec - exiting 19:20:48 (7052): No heartbeat from core client for 30 sec - exiting 19:20:49 (7052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:20:50 (7052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6748, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:37:39 (6300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9024, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 14:32:28 (7212): No heartbeat from core client for 30 sec - exiting 14:32:29 (7212): No heartbeat from core client for 30 sec - exiting 14:32:31 (7212): No heartbeat from core client for 30 sec - exiting 14:32:32 (7212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:57:50 (4824): No heartbeat from core client for 30 sec - exiting 19:57:51 (4824): No heartbeat from core client for 30 sec - exiting 19:57:52 (4824): No heartbeat from core client for 30 sec - exiting 19:57:54 (4824): No heartbeat from core client for 30 sec - exiting 19:57:55 (4824): No heartbeat from core client for 30 sec - exiting 19:57:56 (4824): No heartbeat from core client for 30 sec - exiting 19:57:57 (4824): No heartbeat from core client for 30 sec - exiting 19:57:58 (4824):CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7068, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6336, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3140, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=976, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4196, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4196, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4196, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4196, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4196, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4196, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Oct 2013 18:09:05	967258	15684158	hadcm3n_u2nt_2020_40_008336019_0	233,280	646,938	2.7732
23 Sep 2013 20:38:10	967258	15684158	hadcm3n_u2nt_2020_40_008336019_0	207,360	591,710	2.8535
21 Aug 2013 18:43:56	967258	15684158	hadcm3n_u2nt_2020_40_008336019_0	181,440	534,179	2.9441
02 Jul 2013 11:52:38	967258	15684158	hadcm3n_u2nt_2020_40_008336019_0	155,520	468,512	3.0126
10 Jun 2013 23:54:14	967258	15684158	hadcm3n_u2nt_2020_40_008336019_0	129,600	382,255	2.9495
02 Jun 2013 21:33:56	967258	15684158	hadcm3n_u2nt_2020_40_008336019_0	103,680	301,751	2.9104
16 May 2013 19:27:16	967258	15684158	hadcm3n_u2nt_2020_40_008336019_0	77,760	212,795	2.7366
27 Apr 2013 17:32:35	967258	15684158	hadcm3n_u2nt_2020_40_008336019_0	51,840	136,915	2.6411
18 Apr 2013 16:57:09	967258	15684158	hadcm3n_u2nt_2020_40_008336019_0	25,920	72,804	2.8088