Task 14360347

Name	hadcm3n_o0ry_2020_40_007857236_1
Workunit	8012348
Created	4 Apr 2012, 21:49:42 UTC
Sent	4 Apr 2012, 21:51:47 UTC
Report deadline	5 Jul 2012, 5:18:58 UTC
Received	8 Jun 2012, 11:24:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1148874
Run time	27 days 18 hours 35 min 2 sec
CPU time	25 days 0 hours 8 min 16 sec
Validate state	Invalid
Credit	11,508.48
Device peak FLOPS	1.59 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:06:37 (6872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:06:40 (6872): No heartbeat from core client for 30 sec - exiting 01:08:00 (21667): No heartbeat from core client for 30 sec - exiting 01:08:03 (21667): No heartbeat from core client for 30 sec - exiting 01:08:04 (21667): No heartbeat from core client for 30 sec - exiting 01:08:05 (21667): No heartbeat from core client for 30 sec - exiting 01:08:06 (21667): No heartbeat from core client for 30 sec - exiting 01:08:07 (21667): No heartbeat from core client for 30 sec - exiting 01:08:08 (21667): No heartbeat from core client for 30 sec - exiting 01:08:09 (21667): No heartbeat from core client for 30 sec - exiting 01:08:10 (21667): No heartbeat from core client for 30 sec - exiting 01:08:11 (21667): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:08:12 (21667): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 18:10:57 (27324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 14:37:56 (1200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:28 (1727): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:05:46 (6072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:05:53 (6428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:24 (808): No heartbeat from core client for 30 sec - exiting 02:16:52 (808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:00:45 (7826): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:03:15 (31331): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:59 (5506): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:41:09 (25652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:02:30 (27726): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:23:54 (12884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:00 (12884): No heartbeat from core client for 30 sec - exiting 06:36:47 (24903): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:02:32 (31904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:36:19 (20050): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:36:20 (20050): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 11:10:19 (7648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:43:15 (3147): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:02:45 (27064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:42:08 (21061): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:42:10 (21061): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 17:51:35 (10641): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:11:05 (16176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:47:56 (31950): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:57:30 (29612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:57:31 (29612): No heartbeat from core client for 30 sec - exiting 20:57:32 (29612): No heartbeat from core client for 30 sec - exiting 20:57:33 (29612): No heartbeat from core client for 30 sec - exiting 20:57:34 (29612): No heartbeat from core client for 30 sec - exiting 20:57:35 (29612): No heartbeat from core client for 30 sec - exiting 20:57:36 (29612): No heartbeat from core client for 30 sec - exiting 20:57:37 (29612): No heartbeat from core client for 30 sec - exiting 20:57:38 (29612): No heartbeat from core client for 30 sec - exiting 20:57:39 (29612): No heartbeat from core client for 30 sec - exiting 20:57:40 (29612): No heartbeat from core client for 30 sec - exiting 20:57:41 (29612): No heartbeat from core client for 30 sec - exiting 20:57:42 (29612): No heartbeat from core client for 30 sec - exiting 20:57:43 (29612): No heartbeat from core client for 30 sec - exiting 20:57:44 (29612): No heartbeat from core client for 30 sec - exiting 20:57:45 (29612): No heartbeat from core client for 30 sec - exiting 20:57:46 (29612): No heartbeat from core client for 30 sec - exiting 20:57:47 (29612): No heartbeat from core client for 30 sec - exiting 20:57:48 (29612): No heartbeat from core client for 30 sec - exiting 20:57:49 (29612): No heartbeat from core client for 30 sec - exiting 20:57:50 (29612): No heartbeat from core client for 30 sec - exiting 20:57:51 (29612): No heartbeat from core client for 30 sec - exiting 20:57:52 (29612): No heartbeat from core client for 30 sec - exiting 20:57:53 (29612): No heartbeat from core client for 30 sec - exiting 20:57:54 (29612): No heartbeat from core client for 30 sec - exiting 20:57:55 (29612): No heartbeat from core client for 30 sec - exiting 20:57:56 (29612): No heartbeat from core client for 30 sec - exiting 20:57:57 (29612): No heartbeat from core client for 30 sec - exiting 20:57:58 (29612): No heartbeat from core client for 30 sec - exiting 20:57:59 (29612): No heartbeat from core client for 30 sec - exiting 20:58:00 (29612): No heartbeat from core client for 30 sec - exiting 20:58:01 (29612): No heartbeat from core client for 30 sec - exiting 20:58:02 (29612): No heartbeat from core client for 30 sec - exiting 20:58:03 (29612): No heartbeat from core client for 30 sec - exiting 20:58:04 (29612): No heartbeat from core client for 30 sec - exiting 21:42:54 (16656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:52:37 (10865): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:52:38 (10865): No heartbeat from core client for 30 sec - exiting 21:52:39 (10865): No heartbeat from core client for 30 sec - exiting 21:52:40 (10865): No heartbeat from core client for 30 sec - exiting 21:52:41 (10865): No heartbeat from core client for 30 sec - exiting 21:52:42 (10865): No heartbeat from core client for 30 sec - exiting 21:52:43 (10865): No heartbeat from core client for 30 sec - exiting 21:52:44 (10865): No heartbeat from core client for 30 sec - exiting 21:52:45 (10865): No heartbeat from core client for 30 sec - exiting 21:52:46 (10865): No heartbeat from core client for 30 sec - exiting 21:52:47 (10865): No heartbeat from core client for 30 sec - exiting 21:52:48 (10865): No heartbeat from core client for 30 sec - exiting 21:52:49 (10865): No heartbeat from core client for 30 sec - exiting 23:10:54 (16524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:28:35 (31903): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:03:19 (15636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:58:11 (4206): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:58:14 (4206): No heartbeat from core client for 30 sec - exiting 06:58:15 (4206): No heartbeat from core client for 30 sec - exiting 06:58:16 (4206): No heartbeat from core client for 30 sec - exiting 06:58:17 (4206): No heartbeat from core client for 30 sec - exiting 06:58:18 (4206): No heartbeat from core client for 30 sec - exiting 06:59:06 (11117): No heartbeat from core client for 30 sec - exiting 06:59:07 (11117): No heartbeat from core client for 30 sec - exiting 06:59:08 (11117): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:11 (11525): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:15 (11525): No heartbeat from core client for 30 sec - exiting 07:38:15 (3017): No heartbeat from core client for 30 sec - exiting 07:38:16 (3017): No heartbeat from core client for 30 sec - exiting 07:38:17 (3017): No heartbeat from core client for 30 sec - exiting 07:38:18 (3017): No heartbeat from core client for 30 sec - exiting 07:38:19 (3017): No heartbeat from core client for 30 sec - exiting 07:38:20 (3017): No heartbeat from core client for 30 sec - exiting 07:38:21 (3017): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:28 (3826): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:23 (26197): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:24 (26197): No heartbeat from core client for 30 sec - exiting 11:09:17 (18717): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... 11:48:23 (31749): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:48:25 (31749): No heartbeat from core client for 30 sec - exiting 12:49:21 (20147): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:32:19 (24442): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:34:27 (24442): No heartbeat from core client for 30 sec - exiting 13:34:28 (24442): No heartbeat from core client for 30 sec - exiting 13:34:29 (24442): No heartbeat from core client for 30 sec - exiting 13:34:30 (24442): No heartbeat from core client for 30 sec - exiting 13:34:31 (24442): No heartbeat from core client for 30 sec - exiting 13:34:32 (24442): No heartbeat from core client for 30 sec - exiting 13:34:33 (24442): No heartbeat from core client for 30 sec - exiting 13:34:34 (24442): No heartbeat from core client for 30 sec - exiting 13:34:35 (24442): No heartbeat from core client for 30 sec - exiting 13:34:36 (24442): No heartbeat from core client for 30 sec - exiting 13:34:37 (24442): No heartbeat from core client for 30 sec - exiting 13:34:38 (24442): No heartbeat from core client for 30 sec - exiting 13:34:39 (24442): No heartbeat from core client for 30 sec - exiting 13:34:40 (24442): No heartbeat from core client for 30 sec - exiting 13:34:41 (24442): No heartbeat from core client for 30 sec - exiting 15:02:13 (17180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:38:06 (5077): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:49:53 (31784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 16:58:29 (12023): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:58:30 (12023): No heartbeat from core client for 30 sec - exiting 16:58:31 (12023): No heartbeat from core client for 30 sec - exiting 16:58:32 (12023): No heartbeat from core client for 30 sec - exiting 16:58:33 (12023): No heartbeat from core client for 30 sec - exiting 16:58:34 (12023): No heartbeat from core client for 30 sec - exiting 16:58:35 (12023): No heartbeat from core client for 30 sec - exiting 16:58:36 (12023): No heartbeat from core client for 30 sec - exiting 16:58:37 (12023): No heartbeat from core client for 30 sec - exiting 16:58:38 (12023): No heartbeat from core client for 30 sec - exiting 16:58:39 (12023): No heartbeat from core client for 30 sec - exiting 16:58:40 (12023): No heartbeat from core client for 30 sec - exiting 16:58:41 (12023): No heartbeat from core client for 30 sec - exiting 16:58:42 (12023): No heartbeat from core client for 30 sec - exiting 16:58:43 (12023): No heartbeat from core client for 30 sec - exiting 16:58:44 (12023): No heartbeat from core client for 30 sec - exiting 16:58:45 (12023): No heartbeat from core client for 30 sec - exiting 16:58:46 (12023): No heartbeat from core client for 30 sec - exiting 16:58:47 (12023): No heartbeat from core client for 30 sec - exiting 16:58:48 (12023): No heartbeat from core client for 30 sec - exiting 16:58:49 (12023): No heartbeat from core client for 30 sec - exiting 16:58:50 (12023): No heartbeat from core client for 30 sec - exiting 16:58:51 (12023): No heartbeat from core client for 30 sec - exiting 16:58:52 (12023): No heartbeat from core client for 30 sec - exiting 16:58:53 (12023): No heartbeat from core client for 30 sec - exiting 16:58:54 (12023): No heartbeat from core client for 30 sec - exiting 16:58:55 (12023): No heartbeat from core client for 30 sec - exiting 16:58:56 (12023): No heartbeat from core client for 30 sec - exiting 16:58:57 (12023): No heartbeat from core client for 30 sec - exiting 16:58:58 (12023): No heartbeat from core client for 30 sec - exiting 16:58:59 (12023): No heartbeat from core client for 30 sec - exiting 16:59:00 (12023): No heartbeat from core client for 30 sec - exiting 16:59:01 (12023): No heartbeat from core client for 30 sec - exiting 16:59:02 (12023): No heartbeat from core client for 30 sec - exiting 16:59:03 (12023): No heartbeat from core client for 30 sec - exiting 16:59:04 (12023): No heartbeat from core client for 30 sec - exiting 16:59:05 (12023): No heartbeat from core client for 30 sec - exiting 16:59:06 (12023): No heartbeat from core client for 30 sec - exiting 16:59:07 (12023): No heartbeat from core client for 30 sec - exiting 16:59:08 (12023): No heartbeat from core client for 30 sec - exiting 16:59:09 (12023): No heartbeat from core client for 30 sec - exiting 16:59:10 (12023): No heartbeat from core client for 30 sec - exiting 16:59:11 (12023): No heartbeat from core client for 30 sec - exiting 16:59:12 (12023): No heartbeat from core client for 30 sec - exiting 16:59:13 (12023): No heartbeat from core client for 30 sec - exiting 16:59:14 (12023): No heartbeat from core client for 30 sec - exiting 16:59:15 (12023): No heartbeat from core client for 30 sec - exiting 16:59:16 (12023): No heartbeat from core client for 30 sec - exiting 16:59:17 (12023): No heartbeat from core client for 30 sec - exiting 16:59:18 (12023): No heartbeat from core client for 30 sec - exiting 16:59:19 (12023): No heartbeat from core client for 30 sec - exiting 16:59:20 (12023): No heartbeat from core client for 30 sec - exiting 16:59:21 (12023): No heartbeat from core client for 30 sec - exiting 16:59:22 (12023): No heartbeat from core client for 30 sec - exiting 16:59:23 (12023): No heartbeat from core client for 30 sec - exiting 16:59:24 (12023): No heartbeat from core client for 30 sec - exiting 16:59:25 (12023): No heartbeat from core client for 30 sec - exiting 16:59:26 (12023): No heartbeat from core client for 30 sec - exiting 16:59:27 (12023): No heartbeat from core client for 30 sec - exiting 16:59:28 (12023): No heartbeat from core client for 30 sec - exiting 16:59:29 (12023): No heartbeat from core client for 30 sec - exiting 16:59:30 (12023): No heartbeat from core client for 30 sec - exiting 16:59:31 (12023): No heartbeat from core client for 30 sec - exiting 16:59:32 (12023): No heartbeat from core client for 30 sec - exiting 16:59:33 (12023): No heartbeat from core client for 30 sec - exiting 16:59:34 (12023): No heartbeat from core client for 30 sec - exiting 16:59:35 (12023): No heartbeat from core client for 30 sec - exiting 16:59:36 (12023): No heartbeat from core client for 30 sec - exiting 16:59:37 (12023): No heartbeat from core client for 30 sec - exiting 16:59:38 (12023): No heartbeat from core client for 30 sec - exiting 16:59:39 (12023): No heartbeat from core client for 30 sec - exiting 16:59:40 (12023): No heartbeat from core client for 30 sec - exiting 16:59:41 (12023): No heartbeat from core client for 30 sec - exiting 16:59:42 (12023): No heartbeat from core client for 30 sec - exiting 16:59:43 (12023): No heartbeat from core client for 30 sec - exiting 16:59:44 (12023): No heartbeat from core client for 30 sec - exiting 16:59:45 (12023): No heartbeat from core client for 30 sec - exiting 16:59:46 (12023): No heartbeat from core client for 30 sec - exiting 16:59:47 (12023): No heartbeat from core client for 30 sec - exiting 16:59:48 (12023): No heartbeat from core client for 30 sec - exiting 16:59:49 (12023): No heartbeat from core client for 30 sec - exiting 16:59:50 (12023): No heartbeat from core client for 30 sec - exiting 16:59:51 (12023): No heartbeat from core client for 30 sec - exiting 16:59:52 (12023): No heartbeat from core client for 30 sec - exiting 16:59:53 (12023): No heartbeat from core client for 30 sec - exiting 16:59:54 (12023): No heartbeat from core client for 30 sec - exiting 16:59:55 (12023): No heartbeat from core client for 30 sec - exiting 16:59:56 (12023): No heartbeat from core client for 30 sec - exiting 16:59:57 (12023): No heartbeat from core client for 30 sec - exiting 16:59:58 (12023): No heartbeat from core client for 30 sec - exiting 16:59:59 (12023): No heartbeat from core client for 30 sec - exiting 17:00:00 (12023): No heartbeat from core client for 30 sec - exiting 17:00:01 (12023): No heartbeat from core client for 30 sec - exiting 17:00:02 (12023): No heartbeat from core client for 30 sec - exiting 17:00:03 (12023): No heartbeat from core client for 30 sec - exiting 17:00:04 (12023): No heartbeat from core client for 30 sec - exiting 17:00:05 (12023): No heartbeat from core client for 30 sec - exiting 17:00:06 (12023): No heartbeat from core client for 30 sec - exiting 17:00:07 (12023): No heartbeat from core client for 30 sec - exiting 17:00:08 (12023): No heartbeat from core client for 30 sec - exiting 17:00:09 (12023): No heartbeat from core client for 30 sec - exiting 17:00:10 (12023): No heartbeat from core client for 30 sec - exiting 17:00:11 (12023): No heartbeat from core client for 30 sec - exiting 17:00:12 (12023): No heartbeat from core client for 30 sec - exiting 17:00:13 (12023): No heartbeat from core client for 30 sec - exiting 17:00:14 (12023): No heartbeat from core client for 30 sec - exiting 17:00:15 (12023): No heartbeat from core client for 30 sec - exiting 17:00:16 (12023): No heartbeat from core client for 30 sec - exiting 17:00:17 (12023): No heartbeat from core client for 30 sec - exiting 17:00:18 (12023): No heartbeat from core client for 30 sec - exiting 17:00:19 (12023): No heartbeat from core client for 30 sec - exiting 17:00:20 (12023): No heartbeat from core client for 30 sec - exiting 17:00:21 (12023): No heartbeat from core client for 30 sec - exiting 17:00:22 (12023): No heartbeat from core client for 30 sec - exiting 17:00:23 (12023): No heartbeat from core client for 30 sec - exiting 17:00:24 (12023): No heartbeat from core client for 30 sec - exiting 17:00:25 (12023): No heartbeat from core client for 30 sec - exiting 17:00:26 (12023): No heartbeat from core client for 30 sec - exiting 17:00:27 (12023): No heartbeat from core client for 30 sec - exiting 17:00:28 (12023): No heartbeat from core client for 30 sec - exiting 17:00:29 (12023): No heartbeat from core client for 30 sec - exiting 17:00:30 (12023): No heartbeat from core client for 30 sec - exiting 17:00:31 (12023): No heartbeat from core client for 30 sec - exiting 17:00:32 (12023): No heartbeat from core client for 30 sec - exiting 17:00:33 (12023): No heartbeat from core client for 30 sec - exiting 17:00:34 (12023): No heartbeat from core client for 30 sec - exiting 17:00:35 (12023): No heartbeat from core client for 30 sec - exiting 17:00:36 (12023): No heartbeat from core client for 30 sec - exiting 17:00:37 (12023): No heartbeat from core client for 30 sec - exiting 17:00:38 (12023): No heartbeat from core client for 30 sec - exiting 17:00:39 (12023): No heartbeat from core client for 30 sec - exiting 17:00:40 (12023): No heartbeat from core client for 30 sec - exiting 17:00:41 (12023): No heartbeat from core client for 30 sec - exiting 17:00:42 (12023): No heartbeat from core client for 30 sec - exiting 17:00:44 (12023): No heartbeat from core client for 30 sec - exiting 17:00:45 (12023): No heartbeat from core client for 30 sec - exiting 17:00:46 (12023): No heartbeat from core client for 30 sec - exiting 17:00:47 (12023): No heartbeat from core client for 30 sec - exiting 17:00:48 (12023): No heartbeat from core client for 30 sec - exiting 17:11:13 (17398): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:11:14 (17398): No heartbeat from core client for 30 sec - exiting 17:11:15 (17398): No heartbeat from core client for 30 sec - exiting 17:11:16 (17398): No heartbeat from core client for 30 sec - exiting 17:11:17 (17398): No heartbeat from core client for 30 sec - exiting 17:11:18 (17398): No heartbeat from core client for 30 sec - exiting 17:11:19 (17398): No heartbeat from core client for 30 sec - exiting 17:11:20 (17398): No heartbeat from core client for 30 sec - exiting 17:11:21 (17398): No heartbeat from core client for 30 sec - exiting 17:11:22 (17398): No heartbeat from core client for 30 sec - exiting 17:11:23 (17398): No heartbeat from core client for 30 sec - exiting 17:11:24 (17398): No heartbeat from core client for 30 sec - exiting 17:11:25 (17398): No heartbeat from core client for 30 sec - exiting 17:11:26 (17398): No heartbeat from core client for 30 sec - exiting 17:11:27 (17398): No heartbeat from core client for 30 sec - exiting 17:11:28 (17398): No heartbeat from core client for 30 sec - exiting 17:11:29 (17398): No heartbeat from core client for 30 sec - exiting 17:11:30 (17398): No heartbeat from core client for 30 sec - exiting 17:11:31 (17398): No heartbeat from core client for 30 sec - exiting 17:11:32 (17398): No heartbeat from core client for 30 sec - exiting 17:11:33 (17398): No heartbeat from core client for 30 sec - exiting 17:11:34 (17398): No heartbeat from core client for 30 sec - exiting 17:11:35 (17398): No heartbeat from core client for 30 sec - exiting 17:11:36 (17398): No heartbeat from core client for 30 sec - exiting 17:11:37 (17398): No heartbeat from core client for 30 sec - exiting 17:11:38 (17398): No heartbeat from core client for 30 sec - exiting 17:11:39 (17398): No heartbeat from core client for 30 sec - exiting 17:11:40 (17398): No heartbeat from core client for 30 sec - exiting 17:11:41 (17398): No heartbeat from core client for 30 sec - exiting 17:28:32 (25105): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:28:34 (25105): No heartbeat from core client for 30 sec - exiting 17:33:32 (1615): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:33:33 (1615): No heartbeat from core client for 30 sec - exiting 17:33:34 (1615): No heartbeat from core client for 30 sec - exiting 17:33:35 (1615): No heartbeat from core client for 30 sec - exiting 17:33:36 (1615): No heartbeat from core client for 30 sec - exiting 17:33:37 (1615): No heartbeat from core client for 30 sec - exiting 17:33:38 (1615): No heartbeat from core client for 30 sec - exiting 17:33:39 (1615): No heartbeat from core client for 30 sec - exiting 17:33:41 (1615): No heartbeat from core client for 30 sec - exiting 17:33:42 (1615): No heartbeat from core client for 30 sec - exiting 17:33:43 (1615): No heartbeat from core client for 30 sec - exiting 19:33:57 (4459): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:43:45 (11304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:47:14 (11304): No heartbeat from core client for 30 sec - exiting 19:47:15 (11304): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 23:36:17 (19168): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 15:46:29 (3044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:01:51 (26539): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:32 (6283): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:12:13 (341): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 12:10:21 (31741): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:42:35 (15279): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 15:00:12 (1540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 18:41:25 (16581): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:44:24 (22124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:44:25 (22124): No heartbeat from core client for 30 sec - exiting 18:44:26 (22124): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 20:56:03 (23231): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:02:41 (13252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:47:13 (19933): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:08:11 (30337): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:15:30 (29628): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:00:43 (9718): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:52:07 (26663): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:52:10 (26663): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 08:06:34 (20364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:08:02 (31605): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:08:16 (31605): No heartbeat from core client for 30 sec - exiting 01:08:17 (31605): No heartbeat from core client for 30 sec - exiting 01:08:18 (31605): No heartbeat from core client for 30 sec - exiting 01:08:19 (31605): No heartbeat from core client for 30 sec - exiting 01:08:20 (31605): No heartbeat from core client for 30 sec - exiting 01:08:21 (31605): No heartbeat from core client for 30 sec - exiting 01:08:22 (31605): No heartbeat from core client for 30 sec - exiting 01:08:23 (31605): No heartbeat from core client for 30 sec - exiting 01:08:24 (31605): No heartbeat from core client for 30 sec - exiting 01:08:25 (31605): No heartbeat from core client for 30 sec - exiting 01:08:26 (31605): No heartbeat from core client for 30 sec - exiting 01:08:27 (31605): No heartbeat from core client for 30 sec - exiting 01:08:28 (31605): No heartbeat from core client for 30 sec - exiting 01:08:29 (31605): No heartbeat from core client for 30 sec - exiting 01:08:30 (31605): No heartbeat from core client for 30 sec - exiting 01:08:31 (31605): No heartbeat from core client for 30 sec - exiting 01:08:33 (31605): No heartbeat from core client for 30 sec - exiting 01:08:34 (31605): No heartbeat from core client for 30 sec - exiting 01:08:35 (31605): No heartbeat from core client for 30 sec - exiting 01:08:36 (31605): No heartbeat from core client for 30 sec - exiting 01:08:37 (31605): No heartbeat from core client for 30 sec - exiting 01:08:38 (31605): No heartbeat from core client for 30 sec - exiting 01:08:39 (31605): No heartbeat from core client for 30 sec - exiting 01:08:40 (31605): No heartbeat from core client for 30 sec - exiting 01:08:41 (31605): No heartbeat from core client for 30 sec - exiting 01:08:42 (31605): No heartbeat from core client for 30 sec - exiting 01:08:43 (31605): No heartbeat from core client for 30 sec - exiting 01:08:44 (31605): No heartbeat from core client for 30 sec - exiting 01:08:45 (31605): No heartbeat from core client for 30 sec - exiting 01:08:46 (31605): No heartbeat from core client for 30 sec - exiting 01:08:47 (31605): No heartbeat from core client for 30 sec - exiting 01:08:48 (31605): No heartbeat from core client for 30 sec - exiting 01:08:49 (31605): No heartbeat from core client for 30 sec - exiting 01:08:50 (31605): No heartbeat from core client for 30 sec - exiting 01:08:51 (31605): No heartbeat from core client for 30 sec - exiting 01:08:52 (31605): No heartbeat from core client for 30 sec - exiting 01:08:53 (31605): No heartbeat from core client for 30 sec - exiting 01:08:54 (31605): No heartbeat from core client for 30 sec - exiting 01:08:55 (31605): No heartbeat from core client for 30 sec - exiting 02:45:16 (10703): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:45:21 (10703): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf773a400] [0xf773a430] /lib32/libc.so.6(gsignal+0x51)[0xf75b1c11] /lib32/libc.so.6(abort+0x182)[0xf75b5032] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf759de46] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7714400] [0xf7714430] /lib32/libc.so.6(gsignal+0x51)[0xf758bc11] /lib32/libc.so.6(abort+0x182)[0xf758f032] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7577e46] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77b4400] [0xf77b4430] /lib32/libc.so.6(gsignal+0x51)[0xf762bc11] /lib32/libc.so.6(abort+0x182)[0xf762f032] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7617e46] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77de400] [0xf77de430] /lib32/libc.so.6(gsignal+0x51)[0xf7655c11] /lib32/libc.so.6(abort+0x182)[0xf7659032] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7641e46] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf778f400] [0xf778f430] /lib32/libc.so.6(gsignal+0x51)[0xf7606c11] /lib32/libc.so.6(abort+0x182)[0xf760a032] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75f2e46] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf774f400] [0xf774f430] /lib32/libc.so.6(gsignal+0x51)[0xf75c6c11] /lib32/libc.so.6(abort+0x182)[0xf75ca032] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75b2e46] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jun 2012 02:11:59	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	959,040	2,108,468	2.1985
06 Jun 2012 07:54:38	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	933,120	2,046,988	2.1937
05 Jun 2012 13:32:38	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	907,200	1,985,694	2.1888
04 Jun 2012 19:26:52	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	881,280	1,924,531	2.1838
04 Jun 2012 01:04:08	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	855,360	1,863,331	2.1784
31 May 2012 11:55:27	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	829,440	1,802,762	2.1735
29 May 2012 02:10:53	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	803,520	1,746,217	2.1732
26 May 2012 00:34:40	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	777,600	1,690,429	2.1739
25 May 2012 08:57:56	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	751,680	1,635,774	2.1762
24 May 2012 16:37:09	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	725,760	1,579,628	2.1765
23 May 2012 01:01:48	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	699,840	1,523,612	2.1771
22 May 2012 06:56:25	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	673,920	1,462,939	2.1708
16 May 2012 06:29:37	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	648,000	1,405,565	2.1691
13 May 2012 18:30:55	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	622,080	1,349,202	2.1689
12 May 2012 22:55:51	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	596,160	1,293,713	2.1701
09 May 2012 12:08:14	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	570,240	1,237,627	2.1704
07 May 2012 23:10:55	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	544,320	1,181,100	2.1699
03 May 2012 12:29:32	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	518,400	1,124,436	2.1691
30 Apr 2012 07:53:27	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	492,480	1,068,011	2.1686
27 Apr 2012 13:02:34	1148874	14360347	hadcm3n_o0ry_2020_40_007857236_1	466,560	1,011,546	2.1681