Task 15979444

Name	hadcm3n_o2bt_1980_40_008386995_3
Workunit	8537854
Created	30 Aug 2013, 12:39:38 UTC
Sent	30 Aug 2013, 12:57:08 UTC
Report deadline	29 Nov 2013, 20:24:19 UTC
Received	30 Sep 2013, 2:43:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1254699
Run time	26 days 19 hours 24 min 35 sec
CPU time	11 days 1 hours 20 min 14 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> 09:05:15 (1064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:16 (1064): No heartbeat from core client for 30 sec - exiting 09:05:17 (1064): No heartbeat from core client for 30 sec - exiting 09:05:18 (1064): No heartbeat from core client for 30 sec - exiting 09:05:19 (1064): No heartbeat from core client for 30 sec - exiting 09:05:20 (1064): No heartbeat from core client for 30 sec - exiting 09:05:21 (1064): No heartbeat from core client for 30 sec - exiting 09:05:22 (1064): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 22:08:31 (8172): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 23:29:41 (5900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:29:43 (5900): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:28:12 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:08:48 (6808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:08:49 (6808): No heartbeat from core client for 30 sec - exiting 01:08:50 (6808): No heartbeat from core client for 30 sec - exiting 01:08:57 (7576): Can't acquire lockfile (32) - waiting 35s Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:18:23 (7576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:24 (7576): No heartbeat from core client for 30 sec - exiting 06:18:25 (7576): No heartbeat from core client for 30 sec - exiting 06:18:26 (7576): No heartbeat from core client for 30 sec - exiting 06:18:27 (7576): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 10:26:34 (7112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:26:35 (7112): No heartbeat from core client for 30 sec - exiting 10:26:36 (7112): No heartbeat from core client for 30 sec - exiting 10:26:38 (7112): No heartbeat from core client for 30 sec - exiting 12:10:53 (6148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:10:54 (6148): No heartbeat from core client for 30 sec - exiting 12:11:00 (6576): Can't acquire lockfile (32) - waiting 35s Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:59:32 (6576): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 16:59:33 (6576): No heartbeat from core client for 30 sec - exiting 16:59:34 (6576): No heartbeat from core client for 30 sec - exiting 16:59:35 (6576): No heartbeat from core client for 30 sec - exiting 16:59:36 (6576): No heartbeat from core client for 30 sec - exiting 16:59:37 (6576): No heartbeat from core client for 30 sec - exiting 16:59:38 (6576): No heartbeat from core client for 30 sec - exiting 16:59:39 (6576): No heartbeat from core client for 30 sec - exiting 16:59:40 (6576): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:57:34 (5944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:57:35 (5944): No heartbeat from core client for 30 sec - exiting 08:57:36 (5944): No heartbeat from core client for 30 sec - exiting 00:14:19 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:14:20 (4804): No heartbeat from core client for 30 sec - exiting 13:24:23 (4992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:24 (4992): No heartbeat from core client for 30 sec - exiting 13:24:25 (4992): No heartbeat from core client for 30 sec - exiting 13:31:49 (9684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:45:47 (9912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:13:08 (8804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:17:24 (8412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:39:14 (9932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:37:48 (4368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:37:50 (4368): No heartbeat from core client for 30 sec - exiting 15:39:14 (10084): No heartbeat from core client for 30 sec - exiting 15:39:15 (10084): No heartbeat from core client for 30 sec - exiting 15:39:16 (10084): No heartbeat from core client for 30 sec - exiting 15:39:17 (10084): No heartbeat from core client for 30 sec - exiting 15:39:18 (10084): No heartbeat from core client for 30 sec - exiting 15:39:19 (10084): No heartbeat from core client for 30 sec - exiting 15:39:21 (10084): No heartbeat from core client for 30 sec - exiting 15:39:22 (10084): No heartbeat from core client for 30 sec - exiting 15:39:23 (10084): No heartbeat from core client for 30 sec - exiting 15:39:24 (10084): No heartbeat from core client for 30 sec - exiting 15:39:25 (10084): No heartbeat from core client for 30 sec - exiting 15:39:27 (10084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:09:20 (8972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:09:21 (8972): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 22:21:49 (10164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:21:50 (10164): No heartbeat from core client for 30 sec - exiting 22:21:51 (10164): No heartbeat from core client for 30 sec - exiting 22:21:53 (10164): No heartbeat from core client for 30 sec - exiting 22:21:54 (10164): No heartbeat from core client for 30 sec - exiting 00:17:45 (9520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:17:47 (9520): No heartbeat from core client for 30 sec - exiting 00:17:48 (9520): No heartbeat from core client for 30 sec - exiting 00:17:49 (9520): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:01:55 (2076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:01:56 (2076): No heartbeat from core client for 30 sec - exiting 20:01:57 (2076): No heartbeat from core client for 30 sec - exiting 20:01:58 (2076): No heartbeat from core client for 30 sec - exiting 20:01:59 (2076): No heartbeat from core client for 30 sec - exiting 20:02:00 (2076): No heartbeat from core client for 30 sec - exiting 20:02:01 (2076): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:14:17 (11952): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 20:14:18 (11952): No heartbeat from core client for 30 sec - exiting 20:14:19 (11952): No heartbeat from core client for 30 sec - exiting 20:14:20 (11952): No heartbeat from core client for 30 sec - exiting 20:14:21 (11952): No heartbeat from core client for 30 sec - exiting 20:14:22 (11952): No heartbeat from core client for 30 sec - exiting 20:14:23 (11952): No heartbeat from core client for 30 sec - exiting 20:14:24 (11952): No heartbeat from core client for 30 sec - exiting 20:59:10 (6140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:06:24 (8576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9332, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:11:13 (5672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:11:14 (5672): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5304, iMonCtr=1 Model crash detected, will try to restart... 12:19:21 (1352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:19:22 (1352): No heartbeat from core client for 30 sec - exiting 12:19:23 (1352): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:08:12 (7256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:13 (7256): No heartbeat from core client for 30 sec - exiting 19:08:14 (7256): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 00:50:02 (7244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:50:03 (7244): No heartbeat from core client for 30 sec - exiting 00:50:04 (7244): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:56:53 (8868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:55 (8868): No heartbeat from core client for 30 sec - exiting 18:56:56 (8868): No heartbeat from core client for 30 sec - exiting 18:56:57 (8868): No heartbeat from core client for 30 sec - exiting 18:56:58 (8868): No heartbeat from core client for 30 sec - exiting 18:56:59 (8868): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:39:17 (3844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:14:03 (6136): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 13:14:06 (6136): No heartbeat from core client for 30 sec - exiting 13:21:18 (8792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:12:08 (8708): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 00:12:09 (8708): No heartbeat from core client for 30 sec - exiting 00:12:10 (8708): No heartbeat from core client for 30 sec - exiting 00:12:11 (8708): No heartbeat from core client for 30 sec - exiting 00:12:12 (8708): No heartbeat from core client for 30 sec - exiting 00:12:13 (8708): No heartbeat from core client for 30 sec - exiting 00:12:14 (8708): No heartbeat from core client for 30 sec - exiting 00:12:15 (8708): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:58:16 (22524): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 06:58:17 (22524): No heartbeat from core client for 30 sec - exiting 06:58:18 (22524): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:36:45 (29344): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 20:36:46 (29344): No heartbeat from core client for 30 sec - exiting 20:36:47 (29344): No heartbeat from core client for 30 sec - exiting 20:36:48 (29344): No heartbeat from core client for 30 sec - exiting 20:36:49 (29344): No heartbeat from core client for 30 sec - exiting 20:36:50 (29344): No heartbeat from core client for 30 sec - exiting 20:36:51 (29344): No heartbeat from core client for 30 sec - exiting 20:36:52 (29344): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 22:19:16 (37744): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:24:57 (14216): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 04:05:52 (3628): No heartbeat from core client for 30 sec - exiting 04:05:53 (3628): No heartbeat from core client for 30 sec - exiting 04:05:54 (3628): No heartbeat from core client for 30 sec - exiting 04:05:55 (3628): No heartbeat from core client for 30 sec - exiting 04:05:56 (3628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:49:26 (3964): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 12:49:27 (3964): No heartbeat from core client for 30 sec - exiting 12:49:28 (3964): No heartbeat from core client for 30 sec - exiting 12:49:29 (3964): No heartbeat from core client for 30 sec - exiting 12:49:30 (3964): No heartbeat from core client for 30 sec - exiting 12:49:31 (3964): No heartbeat from core client for 30 sec - exiting 12:49:32 (3964): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:36:26 (7212): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:05:43 (7056): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 21:05:44 (7056): No heartbeat from core client for 30 sec - exiting 21:05:45 (7056): No heartbeat from core client for 30 sec - exiting 21:05:46 (7056): No heartbeat from core client for 30 sec - exiting 21:05:47 (7056): No heartbeat from core client for 30 sec - exiting 21:05:48 (7056): No heartbeat from core client for 30 sec - exiting 21:05:49 (7056): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:40:28 (9932): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:52:19 (8324): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 11:52:20 (8324): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:40:58 (4692): No heartbeat from core client for 30 sec - exiting 15:40:59 (4692): No heartbeat from core client for 30 sec - exiting 15:41:00 (4692): No heartbeat from core client for 30 sec - exiting 15:41:01 (4692): No heartbeat from core client for 30 sec - exiting 15:41:02 (4692): No heartbeat from core client for 30 sec - exiting 15:41:03 (4692): No heartbeat from core client for 30 sec - exiting 15:41:04 (4692): No heartbeat from core client for 30 sec - exiting 15:41:05 (4692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:41:06 (4692): No heartbeat from core client for 30 sec - exiting 15:41:07 (4692): No heartbeat from core client for 30 sec - exiting 15:41:08 (4692): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:30:55 (6388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:39:55 (9760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:39:56 (9760): No heartbeat from core client for 30 sec - exiting 20:39:57 (9760): No heartbeat from core client for 30 sec - exiting 20:39:58 (9760): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:34:59 (8596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:53:47 (11500): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:51:41 (11180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:51:42 (11180): No heartbeat from core client for 30 sec - exiting 23:51:43 (11180): No heartbeat from core client for 30 sec - exiting 23:51:44 (11180): No heartbeat from core client for 30 sec - exiting 23:51:45 (11180): No heartbeat from core client for 30 sec - exiting 23:51:46 (11180): No heartbeat from core client for 30 sec - exiting 23:51:47 (11180): No heartbeat from core client for 30 sec - exiting 23:51:48 (11180): No heartbeat from core client for 30 sec - exiting 23:51:49 (11180): No heartbeat from core client for 30 sec - exiting 23:51:50 (11180): No heartbeat from core client for 30 sec - exiting 23:51:51 (11180): No heartbeat from core client for 30 sec - exiting 23:51:52 (11180): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 02:17:57 (9604): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:37:58 (6920): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 15:37:59 (6920): No heartbeat from core client for 30 sec - exiting 15:38:00 (6920): No heartbeat from core client for 30 sec - exiting 15:38:01 (6920): No heartbeat from core client for 30 sec - exiting 15:38:02 (6920): No heartbeat from core client for 30 sec - exiting 15:38:03 (6920): No heartbeat from core client for 30 sec - exiting 15:38:04 (6920): No heartbeat from core client for 30 sec - exiting 15:38:05 (6920): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:25:52 (508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:25:53 (508): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:31:05 (1420): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 00:31:06 (1420): No heartbeat from core client for 30 sec - exiting 00:31:07 (1420): No heartbeat from core client for 30 sec - exiting 00:31:08 (1420): No heartbeat from core client for 30 sec - exiting 00:31:09 (1420): No heartbeat from core client for 30 sec - exiting 00:31:10 (1420): No heartbeat from core client for 30 sec - exiting 00:31:11 (1420): No heartbeat from core client for 30 sec - exiting 00:56:53 (1224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:27:45 (1452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:27:46 (1452): No heartbeat from core client for 30 sec - exiting 03:27:47 (1452): No heartbeat from core client for 30 sec - exiting 03:27:48 (1452): No heartbeat from core client for 30 sec - exiting 03:27:49 (1452): No heartbeat from core client for 30 sec - exiting 03:27:50 (1452): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:32:16 (7720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:32:17 (7720): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:34:03 (8388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:04 (8388): No heartbeat from core client for 30 sec - exiting 18:34:05 (8388): No heartbeat from core client for 30 sec - exiting 18:34:06 (8388): No heartbeat from core client for 30 sec - exiting 18:34:08 (8388): No heartbeat from core client for 30 sec - exiting 18:34:09 (8388): No heartbeat from core client for 30 sec - exiting 21:13:36 (19192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:41:21 (29256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:41:22 (29256): No heartbeat from core client for 30 sec - exiting 04:41:23 (29256): No heartbeat from core client for 30 sec - exiting 04:41:24 (29256): No heartbeat from core client for 30 sec - exiting 04:41:25 (29256): No heartbeat from core client for 30 sec - exiting 04:41:26 (29256): No heartbeat from core client for 30 sec - exiting 04:41:27 (29256): No heartbeat from core client for 30 sec - exiting 04:57:59 (23160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:58:01 (23160): No heartbeat from core client for 30 sec - exiting 04:58:02 (23160): No heartbeat from core client for 30 sec - exiting 04:58:03 (23160): No heartbeat from core client for 30 sec - exiting 09:06:39 (16860): No heartbeat from core client for 30 sec - exiting 09:06:40 (16860): No heartbeat from core client for 30 sec - exiting 09:06:41 (16860): No heartbeat from core client for 30 sec - exiting 09:06:42 (16860): No heartbeat from core client for 30 sec - exiting 09:06:43 (16860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:06:44 (16860): No heartbeat from core client for 30 sec - exiting 09:06:45 (16860): No heartbeat from core client for 30 sec - exiting 09:06:46 (16860): No heartbeat from core client for 30 sec - exiting 10:23:07 (30900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:54:26 (33468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:54:27 (33468): No heartbeat from core client for 30 sec - exiting 07:54:28 (33468): No heartbeat from core client for 30 sec - exiting 07:54:29 (33468): No heartbeat from core client for 30 sec - exiting 07:54:30 (33468): No heartbeat from core client for 30 sec - exiting 07:54:31 (33468): No heartbeat from core client for 30 sec - exiting 07:54:32 (33468): No heartbeat from core client for 30 sec - exiting 09:30:07 (25804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:00:47 (14412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:00:49 (14412): No heartbeat from core client for 30 sec - exiting 13:00:50 (14412): No heartbeat from core client for 30 sec - exiting 15:54:57 (5016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:14:00 (5136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:14:02 (5136): No heartbeat from core client for 30 sec - exiting 17:14:03 (5136): No heartbeat from core client for 30 sec - exiting 17:14:04 (5136): No heartbeat from core client for 30 sec - exiting 05:39:24 (21572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:39:25 (21572): No heartbeat from core client for 30 sec - exiting 05:39:26 (21572): No heartbeat from core client for 30 sec - exiting 05:39:27 (21572): No heartbeat from core client for 30 sec - exiting 05:39:28 (21572): No heartbeat from core client for 30 sec - exiting 05:39:29 (21572): No heartbeat from core client for 30 sec - exiting 12:15:33 (12304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:15:35 (12304): No heartbeat from core client for 30 sec - exiting 12:15:36 (12304): No heartbeat from core client for 30 sec - exiting 12:15:37 (12304): No heartbeat from core client for 30 sec - exiting 14:48:14 (29464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:01 (15928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:05:08 (30776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:16:41 (19152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:26:28 (34708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:59:47 (17604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:26 (2728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:27 (2728): No heartbeat from core client for 30 sec - exiting 18:31:28 (2728): No heartbeat from core client for 30 sec - exiting 18:31:29 (2728): No heartbeat from core client for 30 sec - exiting 18:31:30 (2728): No heartbeat from core client for 30 sec - exiting 18:31:31 (2728): No heartbeat from core client for 30 sec - exiting 18:31:32 (2728): No heartbeat from core client for 30 sec - exiting 18:31:33 (2728): No heartbeat from core client for 30 sec - exiting 18:31:34 (2728): No heartbeat from core client for 30 sec - exiting 18:31:35 (2728): No heartbeat from core client for 30 sec - exiting 18:31:36 (2728): No heartbeat from core client for 30 sec - exiting 21:21:17 (9880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:21:18 (9880): No heartbeat from core client for 30 sec - exiting 21:21:19 (9880): No heartbeat from core client for 30 sec - exiting 21:21:21 (9880): No heartbeat from core client for 30 sec - exiting 21:21:22 (9880): No heartbeat from core client for 30 sec - exiting 23:24:40 (25896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:15 (10120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:12:25 (30044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:12:26 (30044): No heartbeat from core client for 30 sec - exiting 12:14:00 (23816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:14:01 (23816): No heartbeat from core client for 30 sec - exiting 12:14:02 (23816): No heartbeat from core client for 30 sec - exiting 16:02:42 (3676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:02:43 (3676): No heartbeat from core client for 30 sec - exiting 16:02:44 (3676): No heartbeat from core client for 30 sec - exiting 16:02:45 (3676): No heartbeat from core client for 30 sec - exiting 16:02:46 (3676): No heartbeat from core client for 30 sec - exiting 16:02:47 (3676): No heartbeat from core client for 30 sec - exiting 18:35:19 (4348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:20 (4348): No heartbeat from core client for 30 sec - exiting 18:35:21 (4348): No heartbeat from core client for 30 sec - exiting 18:35:22 (4348): No heartbeat from core client for 30 sec - exiting 18:35:23 (4348): No heartbeat from core client for 30 sec - exiting 18:35:24 (4348): No heartbeat from core client for 30 sec - exiting 05:09:31 (27808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:36:57 (34224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:36:58 (34224): No heartbeat from core client for 30 sec - exiting 06:36:59 (34224): No heartbeat from core client for 30 sec - exiting 06:37:00 (34224): No heartbeat from core client for 30 sec - exiting 06:37:01 (34224): No heartbeat from core client for 30 sec - exiting 06:37:02 (34224): No heartbeat from core client for 30 sec - exiting 06:37:03 (34224): No heartbeat from core client for 30 sec - exiting 15:54:17 (4308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:14:03 (23724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:53:38 (27848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:53:39 (27848): No heartbeat from core client for 30 sec - exiting 02:38:20 (26636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:38:22 (26636): No heartbeat from core client for 30 sec - exiting 02:38:23 (26636): No heartbeat from core client for 30 sec - exiting 02:38:24 (26636): No heartbeat from core client for 30 sec - exiting Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Sep 2013 01:42:03	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	518,400	955,197	1.8426
28 Sep 2013 19:54:37	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	492,480	901,076	1.8297
26 Sep 2013 17:16:04	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	466,560	853,576	1.8295
24 Sep 2013 05:07:28	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	440,640	808,245	1.8343
21 Sep 2013 23:27:30	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	414,720	762,605	1.8388
21 Sep 2013 00:52:59	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	388,800	712,477	1.8325
18 Sep 2013 06:31:49	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	362,880	667,804	1.8403
15 Sep 2013 04:37:46	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	336,960	622,034	1.8460
13 Sep 2013 10:16:24	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	311,040	577,319	1.8561
11 Sep 2013 20:34:34	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	285,120	531,511	1.8642
10 Sep 2013 07:54:05	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	259,200	485,759	1.8741
09 Sep 2013 06:27:32	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	233,280	437,610	1.8759
08 Sep 2013 07:46:48	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	207,360	390,823	1.8848
07 Sep 2013 08:39:03	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	181,440	342,802	1.8893
06 Sep 2013 10:57:47	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	155,520	294,750	1.8953
05 Sep 2013 19:01:59	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	129,600	241,033	1.8598
05 Sep 2013 09:53:23	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	103,680	189,288	1.8257
04 Sep 2013 01:33:34	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	77,760	141,861	1.8243
02 Sep 2013 07:03:25	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	51,840	94,399	1.8210
31 Aug 2013 08:44:43	1254699	15979444	hadcm3n_o2bt_1980_40_008386995_3	25,920	47,207	1.8213