Task 12453938

Name	hadam3p_eu_xvwr_1986_1_007046563_1
Workunit	7249879
Created	31 Dec 2010, 7:05:30 UTC
Sent	31 Dec 2010, 9:48:43 UTC
Report deadline	13 Dec 2011, 15:08:43 UTC
Received	16 Jan 2011, 20:28:31 UTC
Server state	Over
Outcome	No reply
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1105565
Run time	2 days 20 hours 0 min 43 sec
CPU time	2 days 9 hours 39 min 59 sec
Validate state	Invalid
Credit	1,393.37
Device peak FLOPS	3.14 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Europe v6.08 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> 12:49:41 (7000): No heartbeat from core client for 30 sec - exiting 12:49:42 (7000): No heartbeat from core client for 30 sec - exiting 12:49:43 (7000): No heartbeat from core client for 30 sec - exiting 12:49:45 (7000): No heartbeat from core client for 30 sec - exiting 12:49:46 (7000): No heartbeat from core client for 30 sec - exiting 12:49:47 (7000): No heartbeat from core client for 30 sec - exiting 12:49:48 (7000): No heartbeat from core client for 30 sec - exiting 12:49:49 (7000): No heartbeat from core client for 30 sec - exiting 12:49:50 (7000): No heartbeat from core client for 30 sec - exiting 12:49:51 (7000): No heartbeat from core client for 30 sec - exiting 12:49:52 (7000): No heartbeat from core client for 30 sec - exiting 12:49:53 (7000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:10:55 (3472): No heartbeat from core client for 30 sec - exiting 17:10:56 (3472): No heartbeat from core client for 30 sec - exiting 17:10:58 (3472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:11:58 (5464): No heartbeat from core client for 30 sec - exiting 17:11:59 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:42:53 (5836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1972, selfPID=1972, iMonCtr=2 00:24:04 (4184): No heartbeat from core client for 30 sec - exiting 00:24:05 (4184): No heartbeat from core client for 30 sec - exiting 00:24:06 (4184): No heartbeat from core client for 30 sec - exiting 00:24:07 (4184): No heartbeat from core client for 30 sec - exiting 00:24:08 (4184): No heartbeat from core client for 30 sec - exiting 00:24:10 (4184): No heartbeat from core client for 30 sec - exiting 00:24:11 (4184): No heartbeat from core client for 30 sec - exiting 00:24:12 (4184): No heartbeat from core client for 30 sec - exiting 00:24:13 (4184): No heartbeat from core client for 30 sec - exiting 00:24:14 (4184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:24:15 (4184): No heartbeat from core client for 30 sec - exiting 01:26:22 (4988): No heartbeat from core client for 30 sec - exiting 01:26:23 (4988): No heartbeat from core client for 30 sec - exiting 01:26:24 (4988): No heartbeat from core client for 30 sec - exiting 01:26:25 (4988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:53:09 (836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:18 (5952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:20:50 (4944): No heartbeat from core client for 30 sec - exiting 08:20:51 (4944): No heartbeat from core client for 30 sec - exiting 08:20:53 (4944): No heartbeat from core client for 30 sec - exiting 08:20:54 (4944): No heartbeat from core client for 30 sec - exiting 08:20:55 (4944): No heartbeat from core client for 30 sec - exiting 08:20:56 (4944): No heartbeat from core client for 30 sec - exiting 08:20:57 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:07:35 (7524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:10:40 (4932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No Process Handle Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3320, selfPID=7724, iMonCtr=1 No Process Handle Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3320, selfPID=3320, iMonCtr=1 23:50:49 (4276): No heartbeat from core client for 30 sec - exiting 23:50:50 (4276): No heartbeat from core client for 30 sec - exiting 23:50:51 (4276): No heartbeat from core client for 30 sec - exiting 23:50:53 (4276): No heartbeat from core client for 30 sec - exiting 23:50:54 (4276): No heartbeat from core client for 30 sec - exiting 23:50:55 (4276): No heartbeat from core client for 30 sec - exiting 23:50:56 (4276): No heartbeat from core client for 30 sec - exiting 23:50:57 (4276): No heartbeat from core client for 30 sec - exiting 23:50:58 (4276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:52:52 (5700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:06:02 (1900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:10 (6556): No heartbeat from core client for 30 sec - exiting 01:07:11 (6556): No heartbeat from core client for 30 sec - exiting 01:07:12 (6556): No heartbeat from core client for 30 sec - exiting 01:07:13 (6556): No heartbeat from core client for 30 sec - exiting 01:07:14 (6556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:08:16 (4168): No heartbeat from core client for 30 sec - exiting 01:08:17 (4168): No heartbeat from core client for 30 sec - exiting 01:08:18 (4168): No heartbeat from core client for 30 sec - exiting 01:08:19 (4168): No heartbeat from core client for 30 sec - exiting 01:08:20 (4168): No heartbeat from core client for 30 sec - exiting 01:08:22 (4168): No heartbeat from core client for 30 sec - exiting 01:08:23 (4168): No heartbeat from core client for 30 sec - exiting 01:08:24 (4168): No heartbeat from core client for 30 sec - exiting 01:08:25 (4168): No heartbeat from core client for 30 sec - exiting 01:08:26 (4168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:13:10 (7036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6804, selfPID=6804, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4768, selfPID=4768, iMonCtr=2 15:30:00 (8000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4496, selfPID=4496, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7300, selfPID=4404, iMonCtr=1 No Process Handle Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7300, selfPID=7300, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:31:28 (7404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4296, selfPID=4296, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3736, selfPID=3736, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2208, selfPID=2208, iMonCtr=2 CPDN Monitor - Quit request from BOINC... 14:22:31 (3832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5424, selfPID=5424, iMonCtr=2 19:38:37 (828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:41:26 (4392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:41:27 (4392): No heartbeat from core client for 30 sec - exiting 19:42:15 (1260): No heartbeat from core client for 30 sec - exiting 19:42:16 (1260): No heartbeat from core client for 30 sec - exiting 19:42:17 (1260): No heartbeat from core client for 30 sec - exiting 19:42:18 (1260): No heartbeat from core client for 30 sec - exiting 19:42:19 (1260): No heartbeat from core client for 30 sec - exiting 19:42:20 (1260): No heartbeat from core client for 30 sec - exiting 19:42:21 (1260): No heartbeat from core client for 30 sec - exiting 19:42:22 (1260): No heartbeat from core client for 30 sec - exiting 19:42:23 (1260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:42:24 (1260): No heartbeat from core client for 30 sec - exiting 19:43:24 (2212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:56:12 (4840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No Process Handle Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4648, selfPID=2356, iMonCtr=1 No Process Handle Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4648, selfPID=4648, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=852, selfPID=852, iMonCtr=2 CPDN Monitor - Quit request from BOINC... 06:00:27 (176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:01:09 (6020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:01:44 (5644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:02:18 (2180): No heartbeat from core client for 30 sec - exiting 06:02:19 (2180): No heartbeat from core client for 30 sec - exiting 06:02:20 (2180): No heartbeat from core client for 30 sec - exiting 06:02:21 (2180): No heartbeat from core client for 30 sec - exiting 06:02:22 (2180): No heartbeat from core client for 30 sec - exiting 06:02:23 (2180): No heartbeat from core client for 30 sec - exiting 06:02:24 (2180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No Process Handle Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2312, selfPID=6000, iMonCtr=1 No Process Handle Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2312, selfPID=2312, iMonCtr=1 12:22:43 (4596): No heartbeat from core client for 30 sec - exiting 12:22:44 (4596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:24:53 (2392): No heartbeat from core client for 30 sec - exiting 20:24:54 (2392): No heartbeat from core client for 30 sec - exiting 20:24:55 (2392): No heartbeat from core client for 30 sec - exiting 20:24:56 (2392): No heartbeat from core client for 30 sec - exiting 20:24:57 (2392): No heartbeat from core client for 30 sec - exiting 20:24:58 (2392): No heartbeat from core client for 30 sec - exiting 20:24:59 (2392): No heartbeat from core client for 30 sec - exiting 20:25:01 (2392): No heartbeat from core client for 30 sec - exiting 20:25:02 (2392): No heartbeat from core client for 30 sec - exiting 20:25:03 (2392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:18:25 (5224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:06:31 (7796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3492, selfPID=3492, iMonCtr=2 04:01:24 (740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:02:07 (1508): No heartbeat from core client for 30 sec - exiting 04:02:08 (1508): No heartbeat from core client for 30 sec - exiting 04:02:09 (1508): No heartbeat from core client for 30 sec - exiting 04:02:10 (1508): No heartbeat from core client for 30 sec - exiting 04:02:11 (1508): No heartbeat from core client for 30 sec - exiting 04:02:12 (1508): No heartbeat from core client for 30 sec - exiting 04:02:13 (1508): No heartbeat from core client for 30 sec - exiting 04:02:14 (1508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:03:07 (4212): No heartbeat from core client for 30 sec - exiting 04:03:08 (4212): No heartbeat from core client for 30 sec - exiting 04:03:09 (4212): No heartbeat from core client for 30 sec - exiting 04:03:10 (4212): No heartbeat from core client for 30 sec - exiting 04:03:11 (4212): No heartbeat from core client for 30 sec - exiting 04:03:12 (4212): No heartbeat from core client for 30 sec - exiting 04:03:13 (4212): No heartbeat from core client for 30 sec - exiting 04:03:14 (4212): No heartbeat from core client for 30 sec - exiting 04:03:15 (4212): No heartbeat from core client for 30 sec - exiting 04:03:16 (4212): No heartbeat from core client for 30 sec - exiting 04:03:17 (4212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:03:18 (4212): No heartbeat from core client for 30 sec - exiting 04:03:19 (4212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 11:53:04 (7676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:59:37 (6376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4348, selfPID=4348, iMonCtr=2 CPDN Monitor - Quit request from BOINC... 04:25:14 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:25:15 (5428): No heartbeat from core client for 30 sec - exiting 04:25:16 (5428): No heartbeat from core client for 30 sec - exiting 04:25:17 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3332, selfPID=3332, iMonCtr=2 07:53:35 (6608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:27:14 (7768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:16:10 (7964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:16:12 (7964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7652, selfPID=7652, iMonCtr=2 09:42:47 (7484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2172, selfPID=2172, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7180, selfPID=7180, iMonCtr=2 19:25:25 (5236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7288, selfPID=7288, iMonCtr=2 02:54:38 (8024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6756, selfPID=7028, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:06:38 (4056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5004, selfPID=5004, iMonCtr=2 06:05:01 (6028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:05:02 (6028): No heartbeat from core client for 30 sec - exiting 06:05:03 (6028): No heartbeat from core client for 30 sec - exiting 06:05:04 (6028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 09:30:09 (2440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:57:31 (5968): No heartbeat from core client for 30 sec - exiting 16:57:32 (5968): No heartbeat from core client for 30 sec - exiting 16:57:33 (5968): No heartbeat from core client for 30 sec - exiting 16:57:34 (5968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:57:35 (5968): No heartbeat from core client for 30 sec - exiting 16:57:36 (5968): No heartbeat from core client for 30 sec - exiting 17:25:04 (4124): No heartbeat from core client for 30 sec - exiting 17:25:05 (4124): No heartbeat from core client for 30 sec - exiting 17:25:06 (4124): No heartbeat from core client for 30 sec - exiting 17:25:08 (4124): No heartbeat from core client for 30 sec - exiting 17:25:09 (4124): No heartbeat from core client for 30 sec - exiting 17:25:10 (4124): No heartbeat from core client for 30 sec - exiting 17:25:12 (4124): No heartbeat from core client for 30 sec - exiting 17:25:13 (4124): No heartbeat from core client for 30 sec - exiting 17:25:14 (4124): No heartbeat from core client for 30 sec - exiting 17:25:15 (4124): No heartbeat from core client for 30 sec - exiting 17:25:16 (4124): No heartbeat from core client for 30 sec - exiting 17:25:17 (4124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:39:11 (4896): No heartbeat from core client for 30 sec - exiting 06:39:12 (4896): No heartbeat from core client for 30 sec - exiting 06:39:13 (4896): No heartbeat from core client for 30 sec - exiting 06:39:14 (4896): No heartbeat from core client for 30 sec - exiting 06:39:15 (4896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:39:16 (4896): No heartbeat from core client for 30 sec - exiting 06:39:17 (4896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Jan 2011 16:42:58	1105565	12453938	hadam3p_eu_xvwr_1986_1_007046563_1	80,772	190,857	2.3629
15 Jan 2011 16:28:59	1105565	12453938	hadam3p_eu_xvwr_1986_1_007046563_1	80,736	190,342	2.3576
12 Jan 2011 18:15:16	1105565	12453938	hadam3p_eu_xvwr_1986_1_007046563_1	69,216	162,471	2.3473
11 Jan 2011 00:18:50	1105565	12453938	hadam3p_eu_xvwr_1986_1_007046563_1	57,696	135,516	2.3488
09 Jan 2011 12:03:30	1105565	12453938	hadam3p_eu_xvwr_1986_1_007046563_1	46,176	108,675	2.3535
08 Jan 2011 12:03:46	1105565	12453938	hadam3p_eu_xvwr_1986_1_007046563_1	34,691	82,376	2.3746
07 Jan 2011 20:12:47	1105565	12453938	hadam3p_eu_xvwr_1986_1_007046563_1	34,656	81,900	2.3632
03 Jan 2011 16:45:52	1105565	12453938	hadam3p_eu_xvwr_1986_1_007046563_1	23,136	54,481	2.3548
02 Jan 2011 00:45:28	1105565	12453938	hadam3p_eu_xvwr_1986_1_007046563_1	11,616	27,410	2.3597