Task 19317373

Name	hadcm3n_sdns_198012_480_352_010334604_0
Workunit	10334604
Created	26 Feb 2016, 22:56:48 UTC
Sent	27 Feb 2016, 1:10:18 UTC
Report deadline	8 Feb 2017, 6:30:18 UTC
Received	8 Mar 2016, 23:53:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1339455
Run time	10 days 6 hours 1 min 15 sec
CPU time	9 days 10 hours 11 min 41 sec
Validate state	Invalid
Credit	11,819.52
Device peak FLOPS	3.86 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.4.42</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 06:32:55 (3857): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:33:09 (3857): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 06:37:27 (29605): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:40:24 (29779): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:55:18 (29870): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:57:41 (30541): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:00:20 (30614): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:02:44 (30709): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:15 (30788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:10:45 (30892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:10:49 (30892): No heartbeat from core client for 30 sec - exiting 07:10:50 (30892): No heartbeat from core client for 30 sec - exiting 07:10:51 (30892): No heartbeat from core client for 30 sec - exiting 07:10:52 (30892): No heartbeat from core client for 30 sec - exiting 07:10:53 (30892): No heartbeat from core client for 30 sec - exiting 07:10:54 (30892): No heartbeat from core client for 30 sec - exiting 07:14:32 (31034): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:18:00 (31134): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:21:31 (31230): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:25:26 (31331): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:34 (31473): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:32:07 (31575): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:35:45 (31684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:39:19 (31795): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:43:47 (31901): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:47:20 (32019): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:51:23 (32119): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:55:23 (32183): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:57:28 (32271): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:58:48 (32331): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:31 (32347): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:02:58 (32393): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:05:17 (32434): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:08 (32492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:37 (32527): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation 08:12:39 (32577): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:15:10 (32668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:46 (32723): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:18:56 (32762): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:20:42 (32813): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:11 (32847): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:14 (32847): No heartbeat from core client for 30 sec - exiting 08:24:39 (32900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:26:44 (32940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:37 (32987): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:30:53 (33039): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:30:59 (33039): No heartbeat from core client for 30 sec - exiting 08:33:37 (33085): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:33:46 (33085): No heartbeat from core client for 30 sec - exiting 08:36:32 (33146): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:38:20 (33200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:41:15 (33251): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:43:27 (33300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:46:30 (33323): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:49:36 (33354): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:52:34 (33403): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:55 (33449): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:21 (33478): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:03:57 (33506): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:07:12 (33533): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:09:24 (33553): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:51 (33569): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:15:54 (33588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:16:05 (33588): No heartbeat from core client for 30 sec - exiting 09:19:48 (33617): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:22:39 (33651): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:31:31 (33684): No heartbeat from core client for 30 sec - exiting 09:31:48 (33684): No heartbeat from core client for 30 sec - exiting 09:31:53 (33684): No heartbeat from core client for 30 sec - exiting 09:31:54 (33684): No heartbeat from core client for 30 sec - exiting 09:31:55 (33684): No heartbeat from core client for 30 sec - exiting 09:31:56 (33684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:31:57 (33684): No heartbeat from core client for 30 sec - exiting 02:28:26 (335): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:33:28 (29883): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:41:10 (29966): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:57:49 (30070): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:58:53 (30290): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:04:20 (30303): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:08:46 (30401): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:11:19 (30511): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:11:23 (30511): No heartbeat from core client for 30 sec - exiting 03:14:29 (30559): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:14:31 (30559): No heartbeat from core client for 30 sec - exiting 03:14:32 (30559): No heartbeat from core client for 30 sec - exiting 03:14:33 (30559): No heartbeat from core client for 30 sec - exiting 03:14:34 (30559): No heartbeat from core client for 30 sec - exiting 03:14:35 (30559): No heartbeat from core client for 30 sec - exiting 03:14:36 (30559): No heartbeat from core client for 30 sec - exiting 03:14:37 (30559): No heartbeat from core client for 30 sec - exiting 03:14:38 (30559): No heartbeat from core client for 30 sec - exiting 03:16:49 (30628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:18:38 (30680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:21:11 (30737): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:21:14 (30737): No heartbeat from core client for 30 sec - exiting 03:23:31 (30783): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:25:54 (30846): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:28:20 (30891): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:29:14 (30965): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:31:56 (30971): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:34:05 (31054): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:35:07 (31091): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:37:08 (31102): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:40:10 (31187): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:42:51 (31243): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:17 (31303): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:29 (31303): No heartbeat from core client for 30 sec - exiting 03:47:47 (31341): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:50:20 (31391): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:52:54 (31456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:53:04 (31456): No heartbeat from core client for 30 sec - exiting 03:53:05 (31456): No heartbeat from core client for 30 sec - exiting 03:55:17 (31510): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:56:14 (31564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:57:59 (31571): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:00:50 (31632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:01:53 (31682): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:04:18 (31689): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:06:12 (31730): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:08:26 (31747): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:10:54 (31769): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:16 (31769): No heartbeat from core client for 30 sec - exiting 04:14:58 (31818): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:16:51 (31838): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:18:30 (31861): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:13 (31903): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:24:26 (31921): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:26:33 (31948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:26:38 (31948): No heartbeat from core client for 30 sec - exiting 04:29:45 (31970): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:32:49 (32000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:34:36 (32019): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:39:20 (32029): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:42:00 (32050): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:45:00 (32060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:49:12 (32075): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:53:28 (32086): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:57:16 (32093): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:04:21 (32099): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:05:54 (32099):21:14:20 (322): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 22:01:27 (28949): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:11:00 (29554): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:15:35 (29856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:23:36 (30007): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:31:44 (30266): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:37:09 (30525): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:43:22 (30648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:49:15 (30804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:52:31 (30887): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:54:33 (30947): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:57:22 (30998): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:59:57 (31071): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:03:16 (31121): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:05:50 (31206): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:07:57 (31274): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:37 (31348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:13:35 (31420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:16:01 (31455): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:18:39 (31517): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:21:20 (31583): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:23:24 (31641): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:26:21 (31705): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:29:00 (31761): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:31:33 (31832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:31:36 (31832): No heartbeat from core client for 30 sec - exiting 23:34:08 (31881): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:35:10 (31934): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:37:55 (31945): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:38:01 (31945): No heartbeat from core client for 30 sec - exiting 23:40:57 (31967): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:41:04 (31967): No heartbeat from core client for 30 sec - exiting 23:42:12 (31995): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:44:50 (32003): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:47:32 (32048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:50:06 (32091): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:53:49 (32136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:55:17 (32160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:58:14 (32174): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:01:50 (32181): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:06:25 (32227): No heartbeat from core client for 30 sec - exiting 00:06:41 (32227): No heartbeat from core client for 30 sec - exiting 00:06:42 (32227): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:10:01 (32253): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:44 (32269): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:15:44 (32283): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:18:58 (32301): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:22:26 (32314): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:29:15 (32340): No heartbeat from core client for 30 sec - exiting 00:29:38 (32340): No heartbeat from core client for 30 sec - exiting 00:29:42 (32340): No heartbeat from core client for 30 sec - exiting 00:29:54 (32340): No heartbeat from core client for 30 sec - exiting 00:29:59 (32340): No heartbeat from core client for 30 sec - exiting 00:30:02 (32340): No heartbeat from core client for 30 sec - exiting 00:30:08 (32340): No heartbeat from core client for 30 sec - exiting 00:30:10 (32340): No heartbeat from core client for 30 sec - exiting 00:30:11 (32340): No heartbeat from core client for 30 sec - exiting 00:30:14 (32340): No heartbeat from core client for 30 sec - exiting 00:30:16 (32340): No heartbeat from core client for 30 sec - exiting 00:30:19 (32340): No heartbeat from core client for 30 sec - exiting 00:30:23 (32340): No heartbeat from core client for 30 sec - exiting 00:30:27 (32340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:30:30 (32340): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 17:37:24 (333): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:47:27 (29683): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:57:31 (29793): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:35 (29922): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:10:54 (30080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:58 (30124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:19:13 (30124): No heartbeat from core client for 30 sec - exiting 18:21:30 (30204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:28:43 (30227): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:42 (30309): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:41 (30349): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:58 (30378): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:42:04 (30411): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:44 (30446): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:49:46 (30476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:52:55 (30510): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:53:10 (30510): No heartbeat from core client for 30 sec - exiting 18:56:31 (30546): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:01:17 (30584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:04:20 (30622): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:09:23 (30665): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:13:20 (30698): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:16:54 (30729): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:20:47 (30768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:27:47 (30804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:32:02 (30823): No heartbeat from core client for 30 sec - exiting 19:32:19 (30823): No heartbeat from core client for 30 sec - exiting 19:32:23 (30823): No heartbeat from core client for 30 sec - exiting 19:32:26 (30823): No heartbeat from core client for 30 sec - exiting 19:32:27 (30823): No heartbeat from core client for 30 sec - exiting 19:32:29 (30823): No heartbeat from core client for 30 sec - exiting 19:32:30 (30823): No heartbeat from core client for 30 sec - exiting 19:32:31 (30823): No heartbeat from core client for 30 sec - exiting 19:32:32 (30823): No heartbeat from core client for 30 sec - exiting 19:32:33 (30823): No heartbeat from core client for 30 sec - exiting 19:32:34 (30823): No heartbeat from core client for 30 sec - exiting 19:32:35 (30823): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation 19:36:03 (30831): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:41:47 (30837): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x812c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x812c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x102e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x102e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x102e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x102e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x102e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x102e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x102e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x102e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x102e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(332,0xa0cdb1d4) malloc: * error for object 0x102e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug 12:16:33 (332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:49:36 (29399): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:55:01 (29869): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:23:32 (29908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:51 (30285): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:35:26 (30369): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:39:20 (30432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:48:11 (30483): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:51:32 (30578): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:54:22 (30618): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:59:16 (30655): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:01:18 (30693): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:04:08 (30708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:07:36 (30743): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:07:53 (30743): No heartbeat from core client for 30 sec - exiting 14:11:14 (30789): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:14:53 (30828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:15:13 (30828): No heartbeat from core client for 30 sec - exiting 14:19:55 (30865): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:20:45 (30865): No heartbeat from core client for 30 sec - exiting 14:23:56 (30921): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:27:14 (30935): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:28:55 (30951): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:31:18 (30959): No heartbeat from core client for 30 sec - exiting 14:31:32 (30959): No heartbeat from core client for 30 sec - exiting 14:31:33 (30959): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:31:34 (30959): No heartbeat from core client for 30 sec - exiting 14:36:14 (30969): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:38:07 (30969): No heartbeat from core client for 30 sec - exiting 08:34:38 (334): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:39:34 (34325): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:41:45 (34400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:43:17 (34469): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:56 (34518): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:48:02 (34593): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:49:57 (34647): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:52:44 (34712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:54:56 (34779): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:00 (34779): No heartbeat from core client for 30 sec - exiting 08:56:55 (34841): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:58:34 (34896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation 09:00:37 (34945): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:02:49 (35016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:47 (35079): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:08:30 (35122): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:03 (35156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:14:44 (35210): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:14:54 (35210): No heartbeat from core client for 30 sec - exiting 09:17:11 (35263): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:19:25 (35323): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:21:54 (35346): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:24:23 (35377): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:27:18 (35402): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:28:50 (35421): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:32:08 (35437): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:34:31 (35464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:36:16 (35493): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:38:31 (35516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:40:36 (35549): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:43:15 (35588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:45:50 (35617): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:47:46 (35644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:10 (35683): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:53:05 (35711): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:54:43 (35725): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:56:48 (35736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:58:16 (35749): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:00:24 (35758): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:05:26 (35771): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:08:56 (35788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:13:33 (35820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:16:37 (35828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:19:43 (35841): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:23:27 (35850): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Mar 2016 22:03:31	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	984,960	814,322	0.8268
08 Mar 2016 12:54:04	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	959,040	792,553	0.8264
08 Mar 2016 06:16:47	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	933,120	770,459	0.8257
07 Mar 2016 23:46:30	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	907,200	748,870	0.8255
07 Mar 2016 17:28:41	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	881,280	727,592	0.8256
07 Mar 2016 11:07:41	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	855,360	706,251	0.8257
07 Mar 2016 04:44:35	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	829,440	684,816	0.8256
06 Mar 2016 22:23:43	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	803,520	663,460	0.8257
06 Mar 2016 13:03:08	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	777,600	641,675	0.8252
06 Mar 2016 06:36:46	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	751,680	620,146	0.8250
06 Mar 2016 00:14:18	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	725,760	598,787	0.8250
05 Mar 2016 17:51:15	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	699,840	577,378	0.8250
05 Mar 2016 11:26:52	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	673,920	555,910	0.8249
05 Mar 2016 05:08:55	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	648,000	534,443	0.8248
04 Mar 2016 20:11:58	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	622,080	513,225	0.8250
04 Mar 2016 13:49:46	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	596,160	491,941	0.8252
04 Mar 2016 07:28:16	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	570,240	470,520	0.8251
04 Mar 2016 01:09:33	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	544,320	449,195	0.8252
03 Mar 2016 18:46:37	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	518,400	427,814	0.8253
03 Mar 2016 12:26:16	1339455	19317373	hadcm3n_sdns_198012_480_352_010334604_0	492,480	406,352	0.8251