Task 15596801

Name	hadcm3n_4aym_1940_40_008309669_0
Workunit	8460804
Created	7 Feb 2013, 22:00:08 UTC
Sent	7 Feb 2013, 22:13:09 UTC
Report deadline	10 May 2013, 5:40:20 UTC
Received	15 Jun 2013, 8:25:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1204261
Run time	79 days 8 hours 5 min 32 sec
CPU time	70 days 5 hours 49 min 17 sec
Validate state	Invalid
Credit	9,642.24
Device peak FLOPS	1.08 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> 03:55:32 (5868): No heartbeat from core client for 30 sec - exiting 03:55:33 (5868): No heartbeat from core client for 30 sec - exiting 03:55:34 (5868): No heartbeat from core client for 30 sec - exiting 03:55:35 (5868): No heartbeat from core client for 30 sec - exiting 03:55:36 (5868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:51:22 (5584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:36 (5964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3300, iMonCtr=1 Model crash detected, will try to restart... 11:01:03 (5200): No heartbeat from core client for 30 sec - exiting 11:01:04 (5200): No heartbeat from core client for 30 sec - exiting 11:01:05 (5200): No heartbeat from core client for 30 sec - exiting 11:01:06 (5200): No heartbeat from core client for 30 sec - exiting 11:01:07 (5200): No heartbeat from core client for 30 sec - exiting 11:01:08 (5200): No heartbeat from core client for 30 sec - exiting 11:01:09 (5200): No heartbeat from core client for 30 sec - exiting 11:01:10 (5200): No heartbeat from core client for 30 sec - exiting 11:01:11 (5200): No heartbeat from core client for 30 sec - exiting 11:01:12 (5200): No heartbeat from core client for 30 sec - exiting 11:01:14 (5200): No heartbeat from core client for 30 sec - exiting 11:01:15 (5200): No heartbeat from core client for 30 sec - exiting 11:01:16 (5200): No heartbeat from core client for 30 sec - exiting 11:01:17 (5200): No heartbeat from core client for 30 sec - exiting 11:01:18 (5200): No heartbeat from core client for 30 sec - exiting 11:01:19 (5200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:01:20 (5200): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:45:49 (4248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:32:40 (4724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5340, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:41:39 (6892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:25:26 (10012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:19:34 (4820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:55:21 (6860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:53:51 (2652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:37:28 (144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:35:39 (3140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:36:40 (6096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:10:15 (4640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 02:19:24 (4344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:20:21 (6092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:21:25 (5628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:39:40 (6392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:31:42 (6136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:52:26 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:53:50 (9076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Jun 2013 19:36:15	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	803,520	6,036,982	7.5132
12 Jun 2013 11:39:24	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	777,600	5,897,148	7.5838
22 Apr 2013 23:20:53	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	751,680	5,628,083	7.4873
18 Apr 2013 10:16:50	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	725,760	5,293,058	7.2931
14 Apr 2013 13:13:05	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	699,840	4,996,761	7.1399
11 Apr 2013 03:12:18	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	673,920	4,740,539	7.0343
07 Apr 2013 17:06:23	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	648,000	4,481,376	6.9157
04 Apr 2013 15:45:44	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	622,080	4,240,611	6.8168
01 Apr 2013 12:15:43	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	596,160	4,013,722	6.7326
29 Mar 2013 03:23:12	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	570,240	3,791,125	6.6483
26 Mar 2013 11:07:04	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	544,320	3,582,966	6.5825
23 Mar 2013 13:55:15	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	518,400	3,374,404	6.5093
21 Mar 2013 11:24:51	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	492,480	3,199,689	6.4971
19 Mar 2013 11:34:45	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	466,560	3,031,136	6.4968
17 Mar 2013 03:22:48	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	440,640	2,845,880	6.4585
14 Mar 2013 06:19:08	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	414,720	2,652,107	6.3949
11 Mar 2013 19:15:31	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	388,800	2,462,359	6.3332
09 Mar 2013 13:45:07	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	362,880	2,281,996	6.2886
07 Mar 2013 06:55:03	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	336,960	2,111,144	6.2653
05 Mar 2013 00:08:14	1204261	15596801	hadcm3n_4aym_1940_40_008309669_0	311,040	1,939,706	6.2362