Task 15879004

Name	hadcm3n_4lwg_1940_40_008304444_4
Workunit	8455579
Created	3 Jul 2013, 16:22:27 UTC
Sent	3 Jul 2013, 16:57:26 UTC
Report deadline	3 Oct 2013, 0:24:37 UTC
Received	4 Oct 2013, 4:36:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1164992
Run time	32 days 14 hours 11 min 21 sec
CPU time	14 days 21 hours 2 min 13 sec
Validate state	Invalid
Credit	6,842.88
Device peak FLOPS	1.96 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2876, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4588, iMonCtr=1 Model crash detected, will try to restart... 14:03:12 (4308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:03:13 (4308): No heartbeat from core client for 30 sec - exiting 14:03:14 (4308): No heartbeat from core client for 30 sec - exiting 14:03:15 (4308): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5420, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:09:21 (6408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:09:22 (6408): No heartbeat from core client for 30 sec - exiting 09:09:23 (6408): No heartbeat from core client for 30 sec - exiting 09:09:24 (6408): No heartbeat from core client for 30 sec - exiting 09:22:06 (6452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:22:07 (6452): No heartbeat from core client for 30 sec - exiting 09:22:08 (6452): No heartbeat from core client for 30 sec - exiting 09:22:10 (6452): No heartbeat from core client for 30 sec - exiting 21:03:12 (3128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:04:39 (4068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:04:40 (4068): No heartbeat from core client for 30 sec - exiting 09:04:41 (4068): No heartbeat from core client for 30 sec - exiting 09:04:43 (4068): No heartbeat from core client for 30 sec - exiting 09:04:44 (4068): No heartbeat from core client for 30 sec - exiting 09:04:45 (4068): No heartbeat from core client for 30 sec - exiting 09:04:46 (4068): No heartbeat from core client for 30 sec - exiting 09:08:11 (2232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:08:12 (2232): No heartbeat from core client for 30 sec - exiting 09:08:13 (2232): No heartbeat from core client for 30 sec - exiting 09:19:01 (8236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:19:02 (8236): No heartbeat from core client for 30 sec - exiting 09:19:03 (8236): No heartbeat from core client for 30 sec - exiting 11:53:34 (7664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:53:35 (7664): No heartbeat from core client for 30 sec - exiting 11:53:36 (7664): No heartbeat from core client for 30 sec - exiting 12:57:03 (9100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:57:04 (9100): No heartbeat from core client for 30 sec - exiting 20:17:05 (6000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:17:06 (6000): No heartbeat from core client for 30 sec - exiting 20:17:07 (6000): No heartbeat from core client for 30 sec - exiting 20:17:09 (6000): No heartbeat from core client for 30 sec - exiting 21:23:49 (9800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:23:51 (9800): No heartbeat from core client for 30 sec - exiting 01:36:11 (10072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:36:12 (10072): No heartbeat from core client for 30 sec - exiting 04:43:05 (10212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:43:07 (10212): No heartbeat from core client for 30 sec - exiting 04:43:08 (10212): No heartbeat from core client for 30 sec - exiting 04:43:09 (10212): No heartbeat from core client for 30 sec - exiting 04:43:10 (10212): No heartbeat from core client for 30 sec - exiting 09:08:36 (9588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:08:37 (9588): No heartbeat from core client for 30 sec - exiting 09:08:38 (9588): No heartbeat from core client for 30 sec - exiting 09:08:39 (9588): No heartbeat from core client for 30 sec - exiting 09:14:53 (8844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:14:54 (8844): No heartbeat from core client for 30 sec - exiting 09:14:55 (8844): No heartbeat from core client for 30 sec - exiting 09:14:56 (8844): No heartbeat from core client for 30 sec - exiting 09:14:57 (8844): No heartbeat from core client for 30 sec - exiting 14:09:27 (9676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:16:20 (10444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:16:22 (10444): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:16:26 (10956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:16:27 (10956): No heartbeat from core client for 30 sec - exiting 18:16:28 (10956): No heartbeat from core client for 30 sec - exiting 18:16:29 (10956): No heartbeat from core client for 30 sec - exiting 21:29:12 (10672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:29:14 (10672): No heartbeat from core client for 30 sec - exiting 21:29:15 (10672): No heartbeat from core client for 30 sec - exiting 21:29:16 (10672): No heartbeat from core client for 30 sec - exiting 21:29:17 (10672): No heartbeat from core client for 30 sec - exiting 21:29:18 (10672): No heartbeat from core client for 30 sec - exiting 21:29:19 (10672): No heartbeat from core client for 30 sec - exiting 23:41:55 (10444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:41:56 (10444): No heartbeat from core client for 30 sec - exiting 23:41:58 (10444): No heartbeat from core client for 30 sec - exiting 00:42:07 (11048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:42:08 (11048): No heartbeat from core client for 30 sec - exiting 00:42:10 (11048): No heartbeat from core client for 30 sec - exiting 00:42:11 (11048): No heartbeat from core client for 30 sec - exiting 04:07:04 (10688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:07:05 (10688): No heartbeat from core client for 30 sec - exiting 04:07:06 (10688): No heartbeat from core client for 30 sec - exiting 05:43:52 (12164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:43:54 (12164): No heartbeat from core client for 30 sec - exiting 09:02:40 (11284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:02:41 (11284): No heartbeat from core client for 30 sec - exiting 09:07:41 (12368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:07:42 (12368): No heartbeat from core client for 30 sec - exiting 09:25:06 (10996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:25:07 (10996): No heartbeat from core client for 30 sec - exiting 09:25:08 (10996): No heartbeat from core client for 30 sec - exiting 09:25:10 (10996): No heartbeat from core client for 30 sec - exiting 10:12:47 (12544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:06:25 (12744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:06:26 (12744): No heartbeat from core client for 30 sec - exiting 11:13:04 (13020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:13:05 (13020): No heartbeat from core client for 30 sec - exiting 11:13:06 (13020): No heartbeat from core client for 30 sec - exiting 11:13:07 (13020): No heartbeat from core client for 30 sec - exiting 12:06:30 (11788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:06:31 (11788): No heartbeat from core client for 30 sec - exiting 12:06:32 (11788): No heartbeat from core client for 30 sec - exiting 12:54:13 (12984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:54:14 (12984): No heartbeat from core client for 30 sec - exiting 12:54:15 (12984): No heartbeat from core client for 30 sec - exiting 17:30:13 (12436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:15 (12436): No heartbeat from core client for 30 sec - exiting 20:01:25 (13152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:01:26 (13152): No heartbeat from core client for 30 sec - exiting 04:02:29 (13168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:02:30 (13168): No heartbeat from core client for 30 sec - exiting 04:02:31 (13168): No heartbeat from core client for 30 sec - exiting 06:14:41 (12672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:14:42 (12672): No heartbeat from core client for 30 sec - exiting 06:14:44 (12672): No heartbeat from core client for 30 sec - exiting 09:03:50 (12868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:03:51 (12868): No heartbeat from core client for 30 sec - exiting 09:08:29 (14540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:08:31 (14540): No heartbeat from core client for 30 sec - exiting 09:08:32 (14540): No heartbeat from core client for 30 sec - exiting 09:08:33 (14540): No heartbeat from core client for 30 sec - exiting 09:08:34 (14540): No heartbeat from core client for 30 sec - exiting 11:02:50 (12512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:02:51 (12512): No heartbeat from core client for 30 sec - exiting 11:02:52 (12512): No heartbeat from core client for 30 sec - exiting 11:02:53 (12512): No heartbeat from core client for 30 sec - exiting 11:02:54 (12512): No heartbeat from core client for 30 sec - exiting 12:56:38 (13752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:56:39 (13752): No heartbeat from core client for 30 sec - exiting 12:56:40 (13752): No heartbeat from core client for 30 sec - exiting 16:45:00 (13656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:38:34 (14928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:38:35 (14928): No heartbeat from core client for 30 sec - exiting 17:38:36 (14928): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 12:02:45 (916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:46 (916): No heartbeat from core client for 30 sec - exiting 12:19:41 (6496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:19:42 (6496): No heartbeat from core client for 30 sec - exiting 12:19:43 (6496): No heartbeat from core client for 30 sec - exiting 12:19:44 (6496): No heartbeat from core client for 30 sec - exiting 12:19:45 (6496): No heartbeat from core client for 30 sec - exiting 12:23:14 (9292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:23:15 (9292): No heartbeat from core client for 30 sec - exiting 12:23:16 (9292): No heartbeat from core client for 30 sec - exiting 12:54:59 (5116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:55:00 (5116): No heartbeat from core client for 30 sec - exiting 12:55:01 (5116): No heartbeat from core client for 30 sec - exiting 12:55:02 (5116): No heartbeat from core client for 30 sec - exiting 13:39:30 (6628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:39:31 (6628): No heartbeat from core client for 30 sec - exiting 13:39:32 (6628): No heartbeat from core client for 30 sec - exiting 13:39:33 (6628): No heartbeat from core client for 30 sec - exiting 13:39:34 (6628): No heartbeat from core client for 30 sec - exiting 15:58:13 (8368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:58:14 (8368): No heartbeat from core client for 30 sec - exiting 15:58:15 (8368): No heartbeat from core client for 30 sec - exiting 15:58:16 (8368): No heartbeat from core client for 30 sec - exiting 16:15:53 (6696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:15:54 (6696): No heartbeat from core client for 30 sec - exiting 16:15:55 (6696): No heartbeat from core client for 30 sec - exiting 16:15:56 (6696): No heartbeat from core client for 30 sec - exiting 16:22:32 (6964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:04:04 (916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:04:05 (916): No heartbeat from core client for 30 sec - exiting 17:04:06 (916): No heartbeat from core client for 30 sec - exiting 17:04:07 (916): No heartbeat from core client for 30 sec - exiting 17:06:23 (4568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:06:24 (4568): No heartbeat from core client for 30 sec - exiting 20:07:39 (4132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:07:40 (4132): No heartbeat from core client for 30 sec - exiting 20:07:41 (4132): No heartbeat from core client for 30 sec - exiting 20:07:42 (4132): No heartbeat from core client for 30 sec - exiting 20:07:43 (4132): No heartbeat from core client for 30 sec - exiting 21:05:27 (6260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:05:28 (6260): No heartbeat from core client for 30 sec - exiting 21:05:29 (6260): No heartbeat from core client for 30 sec - exiting 00:01:24 (8244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:12:02 (6908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:12:03 (6908): No heartbeat from core client for 30 sec - exiting 01:12:04 (6908): No heartbeat from core client for 30 sec - exiting 02:03:18 (9868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:03:19 (9868): No heartbeat from core client for 30 sec - exiting 02:03:20 (9868): No heartbeat from core client for 30 sec - exiting 02:03:21 (9868): No heartbeat from core client for 30 sec - exiting 02:03:22 (9868): No heartbeat from core client for 30 sec - exiting 03:18:00 (7120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:18:01 (7120): No heartbeat from core client for 30 sec - exiting 03:21:35 (5652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:21:36 (5652): No heartbeat from core client for 30 sec - exiting 03:21:37 (5652): No heartbeat from core client for 30 sec - exiting 03:21:38 (5652): No heartbeat from core client for 30 sec - exiting 03:25:01 (9600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:25:02 (9600): No heartbeat from core client for 30 sec - exiting 03:25:03 (9600): No heartbeat from core client for 30 sec - exiting 03:41:28 (6308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:29 (6308): No heartbeat from core client for 30 sec - exiting 03:44:54 (2148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:44:55 (2148): No heartbeat from core client for 30 sec - exiting 03:44:56 (2148): No heartbeat from core client for 30 sec - exiting 04:35:20 (5920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:35:21 (5920): No heartbeat from core client for 30 sec - exiting 04:52:00 (6876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:52:01 (6876): No heartbeat from core client for 30 sec - exiting 04:52:02 (6876): No heartbeat from core client for 30 sec - exiting 05:01:50 (6204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:01:51 (6204): No heartbeat from core client for 30 sec - exiting 05:01:52 (6204): No heartbeat from core client for 30 sec - exiting 05:01:53 (6204): No heartbeat from core client for 30 sec - exiting 05:01:54 (6204): No heartbeat from core client for 30 sec - exiting 05:05:24 (6364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:40:02 (7908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:40:03 (7908): No heartbeat from core client for 30 sec - exiting 05:40:04 (7908): No heartbeat from core client for 30 sec - exiting 05:40:05 (7908): No heartbeat from core client for 30 sec - exiting 06:00:07 (3880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:00:08 (3880): No heartbeat from core client for 30 sec - exiting 06:00:09 (3880): No heartbeat from core client for 30 sec - exiting 06:00:10 (3880): No heartbeat from core client for 30 sec - exiting 06:00:11 (3880): No heartbeat from core client for 30 sec - exiting 06:10:11 (3612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:20:09 (7164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:20:10 (7164): No heartbeat from core client for 30 sec - exiting 06:20:11 (7164): No heartbeat from core client for 30 sec - exiting 06:20:12 (7164): No heartbeat from core client for 30 sec - exiting 06:26:44 (5804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:26:45 (5804): No heartbeat from core client for 30 sec - exiting 06:26:46 (5804): No heartbeat from core client for 30 sec - exiting 06:26:47 (5804): No heartbeat from core client for 30 sec - exiting 06:30:10 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:50:31 (864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:00:35 (10016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:00:36 (10016): No heartbeat from core client for 30 sec - exiting 07:00:37 (10016): No heartbeat from core client for 30 sec - exiting 07:00:38 (10016): No heartbeat from core client for 30 sec - exiting 07:00:39 (10016): No heartbeat from core client for 30 sec - exiting 07:17:14 (8108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:15 (8108): No heartbeat from core client for 30 sec - exiting 07:27:12 (4384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:27:13 (4384): No heartbeat from core client for 30 sec - exiting 07:47:29 (5680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:17:23 (8412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:17:24 (8412): No heartbeat from core client for 30 sec - exiting 08:30:51 (6688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:41:01 (9916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:41:02 (9916): No heartbeat from core client for 30 sec - exiting 08:41:03 (9916): No heartbeat from core client for 30 sec - exiting 08:41:04 (9916): No heartbeat from core client for 30 sec - exiting 08:41:05 (9916): No heartbeat from core client for 30 sec - exiting 09:03:31 (5352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:03:32 (5352): No heartbeat from core client for 30 sec - exiting 09:03:33 (5352): No heartbeat from core client for 30 sec - exiting 09:03:34 (5352): No heartbeat from core client for 30 sec - exiting 09:05:52 (6544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:53 (6544): No heartbeat from core client for 30 sec - exiting 09:41:12 (4428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:41:13 (4428): No heartbeat from core client for 30 sec - exiting 09:41:14 (4428): No heartbeat from core client for 30 sec - exiting 10:03:20 (5352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:21 (5352): No heartbeat from core client for 30 sec - exiting 10:51:39 (6956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:51:40 (6956): No heartbeat from core client for 30 sec - exiting 10:51:41 (6956): No heartbeat from core client for 30 sec - exiting 11:08:27 (5896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:08:28 (5896): No heartbeat from core client for 30 sec - exiting 11:08:29 (5896): No heartbeat from core client for 30 sec - exiting 11:13:49 (8544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:13:50 (8544): No heartbeat from core client for 30 sec - exiting 11:32:13 (9636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:32:14 (9636): No heartbeat from core client for 30 sec - exiting 11:42:12 (6900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:42:13 (6900): No heartbeat from core client for 30 sec - exiting 11:59:04 (8516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:59:05 (8516): No heartbeat from core client for 30 sec - exiting 11:59:06 (8516): No heartbeat from core client for 30 sec - exiting 11:59:07 (8516): No heartbeat from core client for 30 sec - exiting 12:20:12 (9004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:20:13 (9004): No heartbeat from core client for 30 sec - exiting 13:00:09 (9712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:00:10 (9712): No heartbeat from core client for 30 sec - exiting 13:00:11 (9712): No heartbeat from core client for 30 sec - exiting 13:00:12 (9712): No heartbeat from core client for 30 sec - exiting 13:00:13 (9712): No heartbeat from core client for 30 sec - exiting 13:20:29 (9692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:20:30 (9692): No heartbeat from core client for 30 sec - exiting 13:20:31 (9692): No heartbeat from core client for 30 sec - exiting 14:15:16 (5272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:15:17 (5272): No heartbeat from core client for 30 sec - exiting 14:15:18 (5272): No heartbeat from core client for 30 sec - exiting 14:15:19 (5272): No heartbeat from core client for 30 sec - exiting 14:30:30 (8860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:30:31 (8860): No heartbeat from core client for 30 sec - exiting 15:01:11 (8152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:15:20 (8184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:12:12 (5232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:12:13 (5232): No heartbeat from core client for 30 sec - exiting 16:12:14 (5232): No heartbeat from core client for 30 sec - exiting 16:12:15 (5232): No heartbeat from core client for 30 sec - exiting 16:12:16 (5232): No heartbeat from core client for 30 sec - exiting 16:21:58 (6732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:21:59 (6732): No heartbeat from core client for 30 sec - exiting 16:22:00 (6732): No heartbeat from core client for 30 sec - exiting 16:22:01 (6732): No heartbeat from core client for 30 sec - exiting 16:31:50 (9068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:31:51 (9068): No heartbeat from core client for 30 sec - exiting 16:31:52 (9068): No heartbeat from core client for 30 sec - exiting 16:41:46 (8012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:41:47 (8012): No heartbeat from core client for 30 sec - exiting 16:41:48 (8012): No heartbeat from core client for 30 sec - exiting 16:41:49 (8012): No heartbeat from core client for 30 sec - exiting 16:55:45 (5896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:46 (5896): No heartbeat from core client for 30 sec - exiting 17:11:55 (9968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:11:56 (9968): No heartbeat from core client for 30 sec - exiting 17:11:57 (9968): No heartbeat from core client for 30 sec - exiting 17:11:58 (9968): No heartbeat from core client for 30 sec - exiting 17:31:59 (2888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:32:00 (2888): No heartbeat from core client for 30 sec - exiting 17:32:01 (2888): No heartbeat from core client for 30 sec - exiting 17:42:10 (8736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:42:11 (8736): No heartbeat from core client for 30 sec - exiting 17:52:29 (8104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:52:30 (8104): No heartbeat from core client for 30 sec - exiting 17:52:31 (8104): No heartbeat from core client for 30 sec - exiting 17:52:32 (8104): No heartbeat from core client for 30 sec - exiting 18:02:13 (8216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:14 (8216): No heartbeat from core client for 30 sec - exiting 18:02:15 (8216): No heartbeat from core client for 30 sec - exiting 18:02:16 (8216): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:32:30 (9664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:32:31 (9664): No heartbeat from core client for 30 sec - exiting 18:49:15 (10112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:49:16 (10112): No heartbeat from core client for 30 sec - exiting 18:49:17 (10112): No heartbeat from core client for 30 sec - exiting 18:49:18 (10112): No heartbeat from core client for 30 sec - exiting 18:52:46 (8472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:52:47 (8472): No heartbeat from core client for 30 sec - exiting 18:52:48 (8472): No heartbeat from core client for 30 sec - exiting 19:12:45 (7860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:12:46 (7860): No heartbeat from core client for 30 sec - exiting 19:48:02 (5844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:48:03 (5844): No heartbeat from core client for 30 sec - exiting 19:48:04 (5844): No heartbeat from core client for 30 sec - exiting 20:00:56 (6772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:57 (6772): No heartbeat from core client for 30 sec - exiting 20:00:58 (6772): No heartbeat from core client for 30 sec - exiting 20:00:59 (6772): No heartbeat from core client for 30 sec - exiting 20:03:42 (6128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:43 (6128): No heartbeat from core client for 30 sec - exiting 20:03:44 (6128): No heartbeat from core client for 30 sec - exiting 20:03:45 (6128): No heartbeat from core client for 30 sec - exiting 20:13:34 (1364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:13:35 (1364): No heartbeat from core client for 30 sec - exiting 20:24:05 (5940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:32 (8196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:33 (8196): No heartbeat from core client for 30 sec - exiting 21:01:34 (8196): No heartbeat from core client for 30 sec - exiting 21:01:35 (8196): No heartbeat from core client for 30 sec - exiting 21:01:36 (8196): No heartbeat from core client for 30 sec - exiting 21:08:25 (9832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:08:26 (9832): No heartbeat from core client for 30 sec - exiting 21:18:48 (1272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:18:49 (1272): No heartbeat from core client for 30 sec - exiting 21:33:41 (6860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:33:42 (6860): No heartbeat from core client for 30 sec - exiting 21:43:58 (5908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:43:59 (5908): No heartbeat from core client for 30 sec - exiting 21:44:00 (5908): No heartbeat from core client for 30 sec - exiting 21:50:42 (5216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:50:43 (5216): No heartbeat from core client for 30 sec - exiting 21:50:44 (5216): No heartbeat from core client for 30 sec - exiting 21:50:45 (5216): No heartbeat from core client for 30 sec - exiting 21:54:03 (5272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:54:04 (5272): No heartbeat from core client for 30 sec - exiting 21:54:05 (5272): No heartbeat from core client for 30 sec - exiting 21:54:06 (5272): No heartbeat from core client for 30 sec - exiting 21:54:07 (5272): No heartbeat from core client for 30 sec - exiting 22:04:02 (1896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:04:03 (1896): No heartbeat from core client for 30 sec - exiting 22:11:21 (6248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:37 (9164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:38 (9164): No heartbeat from core client for 30 sec - exiting 22:29:39 (9164): No heartbeat from core client for 30 sec - exiting 23:42:45 (5208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:42:46 (5208): No heartbeat from core client for 30 sec - exiting 23:42:47 (5208): No heartbeat from core client for 30 sec - exiting 23:52:31 (2304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:10:40 (8464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:10:41 (8464): No heartbeat from core client for 30 sec - exiting 00:10:42 (8464): No heartbeat from core client for 30 sec - exiting 00:20:28 (8876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:20:29 (8876): No heartbeat from core client for 30 sec - exiting 00:20:30 (8876): No heartbeat from core client for 30 sec - exiting 00:20:31 (8876): No heartbeat from core client for 30 sec - exiting 00:20:32 (8876): No heartbeat from core client for 30 sec - exiting 00:43:53 (3376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:43:55 (3376): No heartbeat from core client for 30 sec - exiting 01:06:36 (9736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:06:37 (9736): No heartbeat from core client for 30 sec - exiting 01:06:38 (9736): No heartbeat from core client for 30 sec - exiting 01:06:39 (9736): No heartbeat from core client for 30 sec - exiting 01:23:34 (5216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:43:27 (5384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:43:28 (5384): No heartbeat from core client for 30 sec - exiting 02:43:29 (5384): No heartbeat from core client for 30 sec - exiting 02:47:12 (5364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:47:13 (5364): No heartbeat from core client for 30 sec - exiting 02:47:14 (5364): No heartbeat from core client for 30 sec - exiting 02:47:15 (5364): No heartbeat from core client for 30 sec - exiting 03:13:15 (7832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:13:16 (7832): No heartbeat from core client for 30 sec - exiting 03:13:17 (7832): No heartbeat from core client for 30 sec - exiting 03:13:18 (7832): No heartbeat from core client for 30 sec - exiting 03:13:19 (7832): No heartbeat from core client for 30 sec - exiting 03:15:12 (2592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:15:13 (2592): No heartbeat from core client for 30 sec - exiting 03:15:14 (2592): No heartbeat from core client for 30 sec - exiting 03:15:15 (2592): No heartbeat from core client for 30 sec - exiting 03:15:16 (2592): No heartbeat from core client for 30 sec - exiting 03:58:51 (8196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:08:25 (9808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:08:26 (9808): No heartbeat from core client for 30 sec - exiting 04:08:27 (9808): No heartbeat from core client for 30 sec - exiting 04:08:28 (9808): No heartbeat from core client for 30 sec - exiting 04:08:29 (9808): No heartbeat from core client for 30 sec - exiting 04:47:56 (1708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:47:57 (1708): No heartbeat from core client for 30 sec - exiting 04:47:58 (1708): No heartbeat from core client for 30 sec - exiting 04:52:58 (9936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:52:59 (9936): No heartbeat from core client for 30 sec - exiting 04:53:00 (9936): No heartbeat from core client for 30 sec - exiting 04:53:01 (9936): No heartbeat from core client for 30 sec - exiting 04:53:02 (9936): No heartbeat from core client for 30 sec - exiting 04:53:03 (9936): No heartbeat from core client for 30 sec - exiting 05:05:56 (7432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Aug 2013 17:06:26	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	570,240	1,252,247	2.1960
18 Aug 2013 17:06:26	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	544,320	1,197,580	2.2001
18 Aug 2013 17:06:26	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	518,400	1,137,874	2.1950
18 Aug 2013 17:06:26	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	492,480	1,078,175	2.1893
18 Aug 2013 17:06:26	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	466,560	1,018,480	2.1830
18 Aug 2013 17:06:26	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	440,640	958,794	2.1759
18 Aug 2013 17:06:26	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	414,720	899,766	2.1696
30 Jul 2013 09:46:32	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	388,800	840,675	2.1622
29 Jul 2013 12:46:39	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	362,880	782,225	2.1556
29 Jul 2013 12:46:39	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	336,960	725,583	2.1533
29 Jul 2013 12:46:39	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	311,040	666,430	2.1426
26 Jul 2013 04:35:55	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	285,120	607,315	2.1300
23 Jul 2013 20:03:16	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	259,200	549,337	2.1194
23 Jul 2013 19:28:12	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	233,280	495,575	2.1244
23 Jul 2013 17:16:15	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	207,360	441,914	2.1311
23 Jul 2013 14:36:31	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	181,440	387,813	2.1374
23 Jul 2013 14:36:31	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	155,520	332,954	2.1409
23 Jul 2013 14:36:31	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	129,600	277,660	2.1424
23 Jul 2013 14:36:31	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	103,680	221,835	2.1396
10 Jul 2013 12:51:49	1164992	15879004	hadcm3n_4lwg_1940_40_008304444_4	77,760	165,804	2.1323