Task 13026996

Name	hadcm3n_t6f4_1940_40_007316622_0
Workunit	7514052
Created	29 Jun 2011, 4:07:46 UTC
Sent	29 Jun 2011, 15:19:45 UTC
Report deadline	28 Sep 2011, 22:46:56 UTC
Received	14 Aug 2011, 12:58:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1153316
Run time	13 days 4 hours 19 min 45 sec
CPU time	13 days 4 hours 19 min 45 sec
Validate state	Invalid
Credit	8,087.04
Device peak FLOPS	2.66 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>5.10.45</core_client_version> <![CDATA[ <message> too many normally harmless exit(s) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2224, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3152, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 08:15:15 (1540): No heartbeat from core client for 30 sec - exiting 08:15:16 (1540): No heartbeat from core client for 30 sec - exiting 08:15:18 (1540): No heartbeat from core client for 30 sec - exiting 08:15:19 (1540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=584, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4048, iMonCtr=1 Model crash detected, will try to restart... 20:18:56 (288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:20:47 (3920): No heartbeat from core client for 30 sec - exiting 20:20:48 (3920): No heartbeat from core client for 30 sec - exiting 20:20:49 (3920): No heartbeat from core client for 30 sec - exiting 20:20:50 (3920): No heartbeat from core client for 30 sec - exiting 20:20:51 (3920): No heartbeat from core client for 30 sec - exiting 20:20:52 (3920): No heartbeat from core client for 30 sec - exiting 20:20:53 (3920): No heartbeat from core client for 30 sec - exiting 20:20:54 (3920): No heartbeat from core client for 30 sec - exiting 20:20:55 (3920): No heartbeat from core client for 30 sec - exiting 20:20:56 (3920): No heartbeat from core client for 30 sec - exiting 20:20:57 (3920): No heartbeat from core client for 30 sec - exiting 20:20:59 (3920): No heartbeat from core client for 30 sec - exiting 20:21:00 (3920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:22:38 (532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:24:27 (1456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:26:16 (932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:04 (3028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:29:53 (3004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:31:42 (3764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:33:30 (2356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:35:19 (2036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:37:08 (1068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:39:12 (2496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:01 (2608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:42:50 (3204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:46:12 (3344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:48:01 (520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:49:50 (1560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:51:39 (1684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:53:27 (2512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:55:16 (2968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:57:05 (3184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:58:53 (3500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:00:42 (2708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:02:31 (1800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:04:19 (2080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:06:08 (552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:07:57 (2856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:10:00 (3292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:11:50 (3496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:13:39 (3788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:15:43 (3764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:17:32 (2096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:19:21 (1616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:21:09 (2564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:22:26 (228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:23:44 (3736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:25:33 (2728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:27:22 (3120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:29:10 (3744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:30:59 (2504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:47 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:34:36 (2100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:36:25 (3280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:38:13 (3604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:40:17 (1584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:07 (216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:44:11 (1140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:46:00 (2584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:47:49 (1448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:49:38 (3200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:51:26 (2628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:53:15 (3600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:55:04 (3780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:56:21 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:57:38 (3804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:59:27 (3964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:01:16 (476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:03:05 (2100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:04:53 (3516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:06:42 (3096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:04 (2600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:11:53 (2000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:13:57 (2836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:15:46 (3836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:17:35 (2436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:19:24 (1692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:21:12 (3568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:23:01 (3128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:24:50 (2744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:26:38 (1744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:28:27 (552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:30:16 (984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:32:04 (3592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:33:53 (188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:35:42 (3452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:37:30 (2424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:39:34 (1532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:41:24 (2184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:42:57 (3132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:44:14 (3024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:46:03 (3404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:47:52 (2192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:49:09 (1708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:50:26 (2204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:52:16 (1808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:54:04 (1744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:55:53 (3180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:57:41 (1092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:59:30 (1676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:01:19 (3664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:03:07 (828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:04:56 (528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:06:45 (816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:08:33 (3176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:10:37 (544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:12:42 (2620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:14:32 (1456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:16:20 (3692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:18:09 (1688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:58 (488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:21:46 (1660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Jul 2011 06:58:52	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	673,920	1,126,142	1.6710
30 Jul 2011 06:58:52	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	648,000	1,086,413	1.6766
30 Jul 2011 06:58:52	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	622,080	1,046,679	1.6825
30 Jul 2011 06:58:52	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	596,160	1,006,925	1.6890
30 Jul 2011 06:58:52	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	570,240	967,142	1.6960
30 Jul 2011 06:58:52	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	544,320	927,357	1.7037
30 Jul 2011 06:58:52	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	518,400	887,555	1.7121
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	492,480	847,791	1.7215
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	466,560	808,019	1.7319
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	440,640	768,317	1.7436
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	414,720	728,535	1.7567
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	388,800	685,298	1.7626
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	362,880	638,965	1.7608
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	336,960	592,640	1.7588
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	311,040	546,363	1.7566
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	285,120	500,014	1.7537
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	259,200	453,561	1.7498
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	233,280	407,193	1.7455
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	207,360	361,372	1.7427
30 Jul 2011 06:58:51	1153316	13026996	hadcm3n_t6f4_1940_40_007316622_0	181,440	315,638	1.7396