Task 16272672

Name	hadcm3n_7bcf_1980_40_008425986_2
Workunit	8576842
Created	16 Jan 2014, 1:50:03 UTC
Sent	16 Jan 2014, 1:50:30 UTC
Report deadline	20 Jul 2023, 7:10:30 UTC
Received	28 Jan 2014, 13:28:36 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1276511
Run time	10 days 1 hours 43 min 52 sec
CPU time	7 days 23 hours 4 min 33 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.81 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 23:53:32 (10948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:26:03 (2196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:27:01 (2196): No heartbeat from core client for 30 sec - exiting 04:27:02 (2196): No heartbeat from core client for 30 sec - exiting 04:27:03 (2196): No heartbeat from core client for 30 sec - exiting 04:27:04 (2196): No heartbeat from core client for 30 sec - exiting 04:27:05 (2196): No heartbeat from core client for 30 sec - exiting 04:27:06 (2196): No heartbeat from core client for 30 sec - exiting 04:27:07 (2196): No heartbeat from core client for 30 sec - exiting 04:27:43 (12568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 08:54:53 (9916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:46 (7724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:04:33 (6480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:04:34 (6480): No heartbeat from core client for 30 sec - exiting 00:04:35 (6480): No heartbeat from core client for 30 sec - exiting 00:04:36 (6480): No heartbeat from core client for 30 sec - exiting 00:04:37 (6480): No heartbeat from core client for 30 sec - exiting 00:04:38 (6480): No heartbeat from core client for 30 sec - exiting 00:04:39 (6480): No heartbeat from core client for 30 sec - exiting 00:04:40 (6480): No heartbeat from core client for 30 sec - exiting 00:04:41 (6480): No heartbeat from core client for 30 sec - exiting 00:04:42 (6480): No heartbeat from core client for 30 sec - exiting 00:04:43 (6480): No heartbeat from core client for 30 sec - exiting 00:06:08 (7628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:06:13 (7628): No heartbeat from core client for 30 sec - exiting 00:06:14 (7628): No heartbeat from core client for 30 sec - exiting 00:06:15 (7628): No heartbeat from core client for 30 sec - exiting 00:06:16 (7628): No heartbeat from core client for 30 sec - exiting 00:06:56 (7040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:25:24 (12412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:25:32 (12412): No heartbeat from core client for 30 sec - exiting 00:26:11 (7636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:40:21 (3936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:40:23 (3936): No heartbeat from core client for 30 sec - exiting 06:40:24 (3936): No heartbeat from core client for 30 sec - exiting 06:40:25 (3936): No heartbeat from core client for 30 sec - exiting 06:40:26 (3936): No heartbeat from core client for 30 sec - exiting 06:40:27 (3936): No heartbeat from core client for 30 sec - exiting 06:40:28 (3936): No heartbeat from core client for 30 sec - exiting 06:40:29 (3936): No heartbeat from core client for 30 sec - exiting 06:40:30 (3936): No heartbeat from core client for 30 sec - exiting 06:40:31 (3936): No heartbeat from core client for 30 sec - exiting 06:40:32 (3936): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 08:06:12 (7540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:06:13 (7540): No heartbeat from core client for 30 sec - exiting 08:06:14 (7540): No heartbeat from core client for 30 sec - exiting 08:06:15 (7540): No heartbeat from core client for 30 sec - exiting 08:06:16 (7540): No heartbeat from core client for 30 sec - exiting 08:06:17 (7540): No heartbeat from core client for 30 sec - exiting 11:57:15 (7440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:57:16 (7440): No heartbeat from core client for 30 sec - exiting 16:50:18 (11612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:01:51 (6480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:55 (10900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:22:11 (13472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 20:48:26 (13224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:49:37 (2132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:04:10 (8708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:31:22 (13392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:32:06 (13392): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 16:46:04 (13892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:46:05 (13892): No heartbeat from core client for 30 sec - exiting 16:46:06 (13892): No heartbeat from core client for 30 sec - exiting 19:39:17 (7964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:23:05 (4208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:13 (7292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:14 (7292): No heartbeat from core client for 30 sec - exiting 22:27:15 (7292): No heartbeat from core client for 30 sec - exiting 22:27:16 (7292): No heartbeat from core client for 30 sec - exiting 22:27:17 (7292): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 22:40:35 (13848): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 00:52:12 (5856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:55:27 (5880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:28 (5880): No heartbeat from core client for 30 sec - exiting 13:55:29 (5880): No heartbeat from core client for 30 sec - exiting 13:55:30 (5880): No heartbeat from core client for 30 sec - exiting 13:55:31 (5880): No heartbeat from core client for 30 sec - exiting 13:55:32 (5880): No heartbeat from core client for 30 sec - exiting 13:55:33 (5880): No heartbeat from core client for 30 sec - exiting 13:55:34 (5880): No heartbeat from core client for 30 sec - exiting 13:55:35 (5880): No heartbeat from core client for 30 sec - exiting 13:55:36 (5880): No heartbeat from core client for 30 sec - exiting 13:55:37 (5880): No heartbeat from core client for 30 sec - exiting 13:56:14 (2216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:56:27 (2216): No heartbeat from core client for 30 sec - exiting 13:56:28 (2216): No heartbeat from core client for 30 sec - exiting 13:56:29 (2216): No heartbeat from core client for 30 sec - exiting 13:56:30 (2216): No heartbeat from core client for 30 sec - exiting 13:56:31 (2216): No heartbeat from core client for 30 sec - exiting 13:56:32 (2216): No heartbeat from core client for 30 sec - exiting 13:56:33 (2216): No heartbeat from core client for 30 sec - exiting 13:56:34 (2216): No heartbeat from core client for 30 sec - exiting 13:56:35 (2216): No heartbeat from core client for 30 sec - exiting 13:56:36 (2216): No heartbeat from core client for 30 sec - exiting 14:16:52 (7888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:03:12 (7492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:55:09 (7660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:55:59 (7660): No heartbeat from core client for 30 sec - exiting 18:56:35 (7604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:34:31 (6900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:34:32 (6900): No heartbeat from core client for 30 sec - exiting 20:44:22 (2776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:26 (7740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:27 (7740): No heartbeat from core client for 30 sec - exiting 22:06:18 (3040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:19:39 (6680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:26:47 (8268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:27:39 (8268): No heartbeat from core client for 30 sec - exiting 08:27:40 (8268): No heartbeat from core client for 30 sec - exiting 08:27:41 (8268): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:23:18 (8172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:23:57 (8168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:12:58 (6780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:54:17 (7520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:16 (9000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:17 (9000): No heartbeat from core client for 30 sec - exiting 09:12:18 (9000): No heartbeat from core client for 30 sec - exiting 09:15:09 (10192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:15:17 (10192): No heartbeat from core client for 30 sec - exiting 09:15:18 (10192): No heartbeat from core client for 30 sec - exiting 09:15:19 (10192): No heartbeat from core client for 30 sec - exiting 09:15:20 (10192): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 15:49:13 (8432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:49:14 (8432): No heartbeat from core client for 30 sec - exiting 15:49:15 (8432): No heartbeat from core client for 30 sec - exiting 16:25:17 (10216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:26:11 (10216): No heartbeat from core client for 30 sec - exiting 16:26:12 (10216): No heartbeat from core client for 30 sec - exiting 16:26:13 (10216): No heartbeat from core client for 30 sec - exiting 16:26:14 (10216): No heartbeat from core client for 30 sec - exiting 16:26:15 (10216): No heartbeat from core client for 30 sec - exiting 16:26:16 (10216): No heartbeat from core client for 30 sec - exiting 16:26:17 (10216): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 18:33:24 (5764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:33:25 (5764): No heartbeat from core client for 30 sec - exiting 18:34:00 (9920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:11:43 (6216): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 20:19:26 (5432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:19:28 (5432): No heartbeat from core client for 30 sec - exiting 20:19:29 (5432): No heartbeat from core client for 30 sec - exiting 20:19:30 (5432): No heartbeat from core client for 30 sec - exiting 20:19:31 (5432): No heartbeat from core client for 30 sec - exiting 20:19:32 (5432): No heartbeat from core client for 30 sec - exiting 20:19:33 (5432): No heartbeat from core client for 30 sec - exiting 20:19:34 (5432): No heartbeat from core client for 30 sec - exiting 20:19:35 (5432): No heartbeat from core client for 30 sec - exiting 20:19:36 (5432): No heartbeat from core client for 30 sec - exiting 20:19:37 (5432): No heartbeat from core client for 30 sec - exiting 23:10:44 (9912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:10:45 (9912): No heartbeat from core client for 30 sec - exiting 23:10:46 (9912): No heartbeat from core client for 30 sec - exiting 23:10:47 (9912): No heartbeat from core client for 30 sec - exiting 23:10:48 (9912): No heartbeat from core client for 30 sec - exiting 23:10:49 (9912): No heartbeat from core client for 30 sec - exiting 23:12:09 (10012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:40:02 (9812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:40:56 (10208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:40:58 (10208): No heartbeat from core client for 30 sec - exiting 04:40:59 (10208): No heartbeat from core client for 30 sec - exiting 04:41:00 (10208): No heartbeat from core client for 30 sec - exiting 04:41:01 (10208): No heartbeat from core client for 30 sec - exiting 04:41:02 (10208): No heartbeat from core client for 30 sec - exiting 04:41:03 (10208): No heartbeat from core client for 30 sec - exiting 04:41:04 (10208): No heartbeat from core client for 30 sec - exiting 04:41:05 (10208): No heartbeat from core client for 30 sec - exiting 04:41:06 (10208): No heartbeat from core client for 30 sec - exiting 04:41:07 (10208): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:29:25 (3980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:29:26 (3980): No heartbeat from core client for 30 sec - exiting 15:29:27 (3980): No heartbeat from core client for 30 sec - exiting 15:29:28 (3980): No heartbeat from core client for 30 sec - exiting 15:29:29 (3980): No heartbeat from core client for 30 sec - exiting 15:29:30 (3980): No heartbeat from core client for 30 sec - exiting 15:29:31 (3980): No heartbeat from core client for 30 sec - exiting 15:29:32 (3980): No heartbeat from core client for 30 sec - exiting 15:29:33 (3980): No heartbeat from core client for 30 sec - exiting 15:30:13 (7008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:46:40 (8928): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:21:13 (3440): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:54:34 (10924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:56:22 (10788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:57:19 (1080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:57:21 (1080): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10856, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold 13:18:42 (10856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:18:43 (10856): No heartbeat from core client for 30 sec - exiting 13:18:44 (10856): No heartbeat from core client for 30 sec - exiting 13:18:45 (10856): No heartbeat from core client for 30 sec - exiting 13:18:46 (10856): No heartbeat from core client for 30 sec - exiting 13:18:48 (10856): No heartbeat from core client for 30 sec - exiting 13:18:49 (10856): No heartbeat from core client for 30 sec - exiting 13:18:50 (10856): No heartbeat from core client for 30 sec - exiting 13:18:51 (10856): No heartbeat from core client for 30 sec - exiting 16:35:43 (8896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:38:28 (6108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:41:11 (11128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:41:52 (11128): No heartbeat from core client for 30 sec - exiting 16:41:53 (11128): No heartbeat from core client for 30 sec - exiting 16:41:54 (11128): No heartbeat from core client for 30 sec - exiting 16:41:55 (11128): No heartbeat from core client for 30 sec - exiting 16:41:56 (11128): No heartbeat from core client for 30 sec - exiting 16:41:57 (11128): No heartbeat from core client for 30 sec - exiting 16:41:58 (11128): No heartbeat from core client for 30 sec - exiting 16:41:59 (11128): No heartbeat from core client for 30 sec - exiting 16:42:31 (11332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:15 (9808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:05:12 (11580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:06:02 (12212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:16:04 (3760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:16:14 (3760): No heartbeat from core client for 30 sec - exiting 01:16:15 (3760): No heartbeat from core client for 30 sec - exiting 01:16:16 (3760): No heartbeat from core client for 30 sec - exiting 01:16:17 (3760): No heartbeat from core client for 30 sec - exiting 01:16:18 (3760): No heartbeat from core client for 30 sec - exiting 01:16:19 (3760): No heartbeat from core client for 30 sec - exiting 01:16:20 (3760): No heartbeat from core client for 30 sec - exiting 01:16:21 (3760): No heartbeat from core client for 30 sec - exiting 04:48:27 (9972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:48:52 (9972): No heartbeat from core client for 30 sec - exiting 04:48:53 (9972): No heartbeat from core client for 30 sec - exiting 04:48:54 (9972): No heartbeat from core client for 30 sec - exiting 04:48:55 (9972): No heartbeat from core client for 30 sec - exiting 05:02:35 (11688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:02:36 (11688): No heartbeat from core client for 30 sec - exiting 05:02:37 (11688): No heartbeat from core client for 30 sec - exiting 05:02:38 (11688): No heartbeat from core client for 30 sec - exiting 05:02:39 (11688): No heartbeat from core client for 30 sec - exiting 05:02:40 (11688): No heartbeat from core client for 30 sec - exiting 05:02:41 (11688): No heartbeat from core client for 30 sec - exiting 05:02:42 (11688): No heartbeat from core client for 30 sec - exiting 05:02:43 (11688): No heartbeat from core client for 30 sec - exiting 05:02:44 (11688): No heartbeat from core client for 30 sec - exiting 05:02:45 (11688): No heartbeat from core client for 30 sec - exiting 05:03:53 (11400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:00:57 (7068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:53:43 (6480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:53:44 (6480): No heartbeat from core client for 30 sec - exiting 12:53:45 (6480): No heartbeat from core client for 30 sec - exiting 12:53:46 (6480): No heartbeat from core client for 30 sec - exiting 12:53:47 (6480): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:59:04 (7884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:59:05 (7884): No heartbeat from core client for 30 sec - exiting 13:59:06 (7884): No heartbeat from core client for 30 sec - exiting 13:59:07 (7884): No heartbeat from core client for 30 sec - exiting 13:59:08 (7884): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:11:11 (3116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:12 (3116): No heartbeat from core client for 30 sec - exiting 23:11:13 (3116): No heartbeat from core client for 30 sec - exiting 23:11:14 (3116): No heartbeat from core client for 30 sec - exiting 23:11:15 (3116): No heartbeat from core client for 30 sec - exiting 23:12:01 (3116): No heartbeat from core client for 30 sec - exiting 23:12:02 (3116): No heartbeat from core client for 30 sec - exiting 23:12:03 (3116): No heartbeat from core client for 30 sec - exiting 23:12:04 (3116): No heartbeat from core client for 30 sec - exiting 23:12:05 (3116): No heartbeat from core client for 30 sec - exiting 23:12:06 (3116): No heartbeat from core client for 30 sec - exiting 23:12:07 (3116): No heartbeat from core client for 30 sec - exiting 23:12:08 (3116): No heartbeat from core client for 30 sec - exiting 23:12:09 (3116): No heartbeat from core client for 30 sec - exiting 23:12:10 (3116): No heartbeat from core client for 30 sec - exiting 23:12:11 (3116): No heartbeat from core client for 30 sec - exiting 23:12:12 (3116): No heartbeat from core client for 30 sec - exiting 23:12:13 (3116): No heartbeat from core client for 30 sec - exiting 23:12:14 (3116): No heartbeat from core client for 30 sec - exiting 23:12:15 (3116): No heartbeat from core client for 30 sec - exiting 23:12:16 (3116): No heartbeat from core client for 30 sec - exiting 23:12:17 (3116): No heartbeat from core client for 30 sec - exiting 23:12:18 (3116): No heartbeat from core client for 30 sec - exiting 23:12:21 (3116): No heartbeat from core client for 30 sec - exiting 23:12:22 (3116): No heartbeat from core client for 30 sec - exiting 23:12:25 (3116): No heartbeat from core client for 30 sec - exiting 23:12:26 (3116): No heartbeat from core client for 30 sec - exiting 23:12:27 (3116): No heartbeat from core client for 30 sec - exiting 23:12:28 (3116): No heartbeat from core client for 30 sec - exiting 23:12:29 (3116): No heartbeat from core client for 30 sec - exiting 23:12:30 (3116): No heartbeat from core client for 30 sec - exiting 23:12:31 (3116): No heartbeat from core client for 30 sec - exiting 23:12:32 (3116): No heartbeat from core client for 30 sec - exiting 23:12:33 (3116): No heartbeat from core client for 30 sec - exiting 23:12:34 (3116): No heartbeat from core client for 30 sec - exiting 23:12:35 (3116): No heartbeat from core client for 30 sec - exiting 23:12:36 (3116): No heartbeat from core client for 30 sec - exiting 01:27:18 (7644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:55:27 (11912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:55:28 (11912): No heartbeat from core client for 30 sec - exiting 06:55:29 (11912): No heartbeat from core client for 30 sec - exiting 06:55:30 (11912): No heartbeat from core client for 30 sec - exiting 06:55:31 (11912): No heartbeat from core client for 30 sec - exiting 06:55:32 (11912): No heartbeat from core client for 30 sec - exiting 06:55:33 (11912): No heartbeat from core client for 30 sec - exiting 06:55:34 (11912): No heartbeat from core client for 30 sec - exiting 06:55:35 (11912): No heartbeat from core client for 30 sec - exiting 06:55:36 (11912): No heartbeat from core client for 30 sec - exiting 06:55:37 (11912): No heartbeat from core client for 30 sec - exiting 06:56:13 (9252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:59:51 (11260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:27:12 (7508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:27:13 (7508): No heartbeat from core client for 30 sec - exiting 08:41:40 (11408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:41:41 (11408): No heartbeat from core client for 30 sec - exiting 08:41:42 (11408): No heartbeat from core client for 30 sec - exiting 08:41:43 (11408): No heartbeat from core client for 30 sec - exiting 08:41:44 (11408): No heartbeat from core client for 30 sec - exiting 08:41:45 (11408): No heartbeat from core client for 30 sec - exiting 08:41:46 (11408): No heartbeat from core client for 30 sec - exiting 08:41:47 (11408): No heartbeat from core client for 30 sec - exiting 08:41:48 (11408): No heartbeat from core client for 30 sec - exiting 08:41:49 (11408): No heartbeat from core client for 30 sec - exiting 08:41:50 (11408): No heartbeat from core client for 30 sec - exiting 08:42:34 (9336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:18:37 (6796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:18:47 (6796): No heartbeat from core client for 30 sec - exiting 10:18:48 (6796): No heartbeat from core client for 30 sec - exiting 10:18:49 (6796): No heartbeat from core client for 30 sec - exiting 10:18:50 (6796): No heartbeat from core client for 30 sec - exiting 10:22:24 (7464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:50:50 (7916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:50:51 (7916): No heartbeat from core client for 30 sec - exiting 12:50:52 (7916): No heartbeat from core client for 30 sec - exiting 12:50:53 (7916): No heartbeat from core client for 30 sec - exiting 12:50:54 (7916): No heartbeat from core client for 30 sec - exiting 12:50:55 (7916): No heartbeat from core client for 30 sec - exiting 12:50:56 (7916): No heartbeat from core client for 30 sec - exiting 12:50:57 (7916): No heartbeat from core client for 30 sec - exiting 12:50:58 (7916): No heartbeat from core client for 30 sec - exiting 12:50:59 (7916): No heartbeat from core client for 30 sec - exiting 12:51:00 (7916): No heartbeat from core client for 30 sec - exiting 13:35:21 (11328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:35:22 (11328): No heartbeat from core client for 30 sec - exiting 13:35:23 (11328): No heartbeat from core client for 30 sec - exiting 13:35:24 (11328): No heartbeat from core client for 30 sec - exiting 13:35:25 (11328): No heartbeat from core client for 30 sec - exiting 13:35:26 (11328): No heartbeat from core client for 30 sec - exiting 13:35:27 (11328): No heartbeat from core client for 30 sec - exiting 13:35:28 (11328): No heartbeat from core client for 30 sec - exiting 13:35:29 (11328): No heartbeat from core client for 30 sec - exiting 13:35:30 (11328): No heartbeat from core client for 30 sec - exiting 13:35:31 (11328): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 13:50:45 (7648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77E23AC3 read attempt to address 0x40DF8096 Engaging BOINC Windows Runtime Debugger... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5024, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5024, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5024, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5024, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Jan 2014 06:15:47	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	518,400	687,247	1.3257
27 Jan 2014 12:33:18	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	492,480	652,868	1.3257
27 Jan 2014 00:31:07	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	466,560	618,995	1.3267
26 Jan 2014 09:37:10	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	440,640	583,858	1.3250
25 Jan 2014 21:29:48	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	414,720	550,252	1.3268
25 Jan 2014 08:07:02	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	388,800	515,707	1.3264
24 Jan 2014 13:38:19	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	362,880	481,285	1.3263
24 Jan 2014 10:24:32	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	336,960	445,541	1.3222
24 Jan 2014 10:24:32	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	311,040	411,558	1.3232
22 Jan 2014 00:16:47	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	285,120	377,651	1.3245
21 Jan 2014 10:50:20	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	259,200	343,981	1.3271
20 Jan 2014 21:47:41	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	233,280	307,688	1.3190
20 Jan 2014 07:40:05	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	207,360	274,185	1.3223
19 Jan 2014 19:17:16	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	181,440	240,364	1.3248
19 Jan 2014 06:44:41	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	155,520	206,141	1.3255
18 Jan 2014 16:30:40	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	129,600	172,505	1.3311
18 Jan 2014 04:31:06	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	103,680	138,167	1.3326
17 Jan 2014 14:38:36	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	77,760	103,752	1.3343
17 Jan 2014 01:35:24	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	51,840	67,907	1.3099
16 Jan 2014 14:21:45	1276511	16272672	hadcm3n_7bcf_1980_40_008425986_2	25,920	34,370	1.3260