Task 14788932

Name	hadcm3n_y96l_1980_40_008000417_0
Workunit	8155531
Created	12 Jun 2012, 22:36:49 UTC
Sent	12 Jun 2012, 22:36:52 UTC
Report deadline	12 Sep 2012, 6:04:03 UTC
Received	24 Jun 2012, 22:59:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1122757
Run time	11 days 4 hours 17 min 41 sec
CPU time	9 days 18 hours 27 min 17 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	1.70 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 04:11:12 (4820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:25 (4820): No heartbeat from core client for 30 sec - exiting 04:11:26 (4820): No heartbeat from core client for 30 sec - exiting 04:11:27 (4820): No heartbeat from core client for 30 sec - exiting 04:11:28 (4820): No heartbeat from core client for 30 sec - exiting 04:11:29 (4820): No heartbeat from core client for 30 sec - exiting 04:11:31 (4820): No heartbeat from core client for 30 sec - exiting 04:11:32 (4820): No heartbeat from core client for 30 sec - exiting 04:11:33 (4820): No heartbeat from core client for 30 sec - exiting 04:11:34 (4820): No heartbeat from core client for 30 sec - exiting 04:11:35 (4820): No heartbeat from core client for 30 sec - exiting 04:11:36 (4820): No heartbeat from core client for 30 sec - exiting 07:16:37 (5116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:27 (6128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:32 (6128): No heartbeat from core client for 30 sec - exiting 07:17:33 (6128): No heartbeat from core client for 30 sec - exiting 07:17:34 (6128): No heartbeat from core client for 30 sec - exiting 07:17:35 (6128): No heartbeat from core client for 30 sec - exiting 14:02:27 (7152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:03:22 (7152): No heartbeat from core client for 30 sec - exiting 14:03:23 (7152): No heartbeat from core client for 30 sec - exiting 14:03:24 (7152): No heartbeat from core client for 30 sec - exiting 14:03:25 (7152): No heartbeat from core client for 30 sec - exiting 14:03:26 (7152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 01:58:33 (2580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:37 (2580): No heartbeat from core client for 30 sec - exiting 02:14:50 (2668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... 00:06:14 (4968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:06:57 (4968): No heartbeat from core client for 30 sec - exiting 00:06:58 (4968): No heartbeat from core client for 30 sec - exiting 00:06:59 (4968): No heartbeat from core client for 30 sec - exiting 00:07:00 (4968): No heartbeat from core client for 30 sec - exiting 00:17:37 (8124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:02:01 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:02:21 (4700): No heartbeat from core client for 30 sec - exiting 02:02:22 (4700): No heartbeat from core client for 30 sec - exiting 02:02:23 (4700): No heartbeat from core client for 30 sec - exiting 02:02:24 (4700): No heartbeat from core client for 30 sec - exiting 02:02:25 (4700): No heartbeat from core client for 30 sec - exiting 02:02:26 (4700): No heartbeat from core client for 30 sec - exiting 02:02:27 (4700): No heartbeat from core client for 30 sec - exiting 02:24:13 (6276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 05:22:29 (5644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:22:30 (5644): No heartbeat from core client for 30 sec - exiting 02:04:49 (7020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:05:13 (7020): No heartbeat from core client for 30 sec - exiting 02:05:14 (7020): No heartbeat from core client for 30 sec - exiting 02:05:15 (7020): No heartbeat from core client for 30 sec - exiting 02:05:16 (7020): No heartbeat from core client for 30 sec - exiting 02:05:17 (7020): No heartbeat from core client for 30 sec - exiting 02:05:18 (7020): No heartbeat from core client for 30 sec - exiting 02:05:19 (7020): No heartbeat from core client for 30 sec - exiting 02:05:20 (7020): No heartbeat from core client for 30 sec - exiting 02:05:21 (7020): No heartbeat from core client for 30 sec - exiting 07:20:52 (3048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:49 (1912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:50 (1912): No heartbeat from core client for 30 sec - exiting 07:28:51 (1912): No heartbeat from core client for 30 sec - exiting 07:28:53 (1912): No heartbeat from core client for 30 sec - exiting 07:28:54 (1912): No heartbeat from core client for 30 sec - exiting 07:52:38 (6748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:48 (6748): No heartbeat from core client for 30 sec - exiting 07:52:49 (6748): No heartbeat from core client for 30 sec - exiting 07:52:50 (6748): No heartbeat from core client for 30 sec - exiting 07:58:03 (2412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:25:10 (1104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:25:16 (1104): No heartbeat from core client for 30 sec - exiting 12:25:17 (1104): No heartbeat from core client for 30 sec - exiting 14:44:50 (3788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:41:47 (500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:27:57 (5824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:05 (3632): No heartbeat from core client for 30 sec - exiting 03:30:17 (3632): No heartbeat from core client for 30 sec - exiting 03:30:18 (3632): No heartbeat from core client for 30 sec - exiting 03:30:19 (3632): No heartbeat from core client for 30 sec - exiting 03:30:21 (3632): No heartbeat from core client for 30 sec - exiting 03:30:22 (3632): No heartbeat from core client for 30 sec - exiting 03:30:23 (3632): No heartbeat from core client for 30 sec - exiting 03:30:24 (3632): No heartbeat from core client for 30 sec - exiting 03:30:25 (3632): No heartbeat from core client for 30 sec - exiting 03:30:26 (3632): No heartbeat from core client for 30 sec - exiting 03:30:27 (3632): No heartbeat from core client for 30 sec - exiting 03:30:28 (3632): No heartbeat from core client for 30 sec - exiting 03:30:29 (3632): No heartbeat from core client for 30 sec - exiting 03:30:30 (3632): No heartbeat from core client for 30 sec - exiting 03:30:31 (3632): No heartbeat from core client for 30 sec - exiting 03:30:33 (3632): No heartbeat from core client for 30 sec - exiting 03:30:34 (3632): No heartbeat from core client for 30 sec - exiting 03:30:35 (3632): No heartbeat from core client for 30 sec - exiting 03:30:36 (3632): No heartbeat from core client for 30 sec - exiting 03:30:37 (3632): No heartbeat from core client for 30 sec - exiting 03:30:38 (3632): No heartbeat from core client for 30 sec - exiting 03:30:39 (3632): No heartbeat from core client for 30 sec - exiting 03:30:40 (3632): No heartbeat from core client for 30 sec - exiting 03:30:41 (3632): No heartbeat from core client for 30 sec - exiting 03:30:42 (3632): No heartbeat from core client for 30 sec - exiting 03:30:43 (3632): No heartbeat from core client for 30 sec - exiting 03:30:45 (3632): No heartbeat from core client for 30 sec - exiting 03:34:56 (3632): No heartbeat from core client for 30 sec - exiting 03:34:57 (3632): No heartbeat from core client for 30 sec - exiting 03:34:58 (3632): No heartbeat from core client for 30 sec - exiting 03:34:59 (3632): No heartbeat from core client for 30 sec - exiting 03:35:00 (3632): No heartbeat from core client for 30 sec - exiting 03:35:01 (3632): No heartbeat from core client for 30 sec - exiting 03:35:02 (3632): No heartbeat from core client for 30 sec - exiting 03:35:03 (3632): No heartbeat from core client for 30 sec - exiting 03:35:04 (3632): No heartbeat from core client for 30 sec - exiting 03:35:06 (3632): No heartbeat from core client for 30 sec - exiting 03:35:07 (3632): No heartbeat from core client for 30 sec - exiting 03:35:08 (3632): No heartbeat from core client for 30 sec - exiting 03:35:09 (3632): No heartbeat from core client for 30 sec - exiting 03:35:10 (3632): No heartbeat from core client for 30 sec - exiting 03:35:11 (3632): No heartbeat from core client for 30 sec - exiting 03:35:12 (3632): No heartbeat from core client for 30 sec - exiting 03:35:13 (3632): No heartbeat from core client for 30 sec - exiting 03:35:14 (3632): No heartbeat from core client for 30 sec - exiting 03:35:15 (3632): No heartbeat from core client for 30 sec - exiting 03:35:16 (3632): No heartbeat from core client for 30 sec - exiting 03:35:18 (3632): No heartbeat from core client for 30 sec - exiting 03:35:19 (3632): No heartbeat from core client for 30 sec - exiting 03:35:20 (3632): No heartbeat from core client for 30 sec - exiting 03:35:21 (3632): No heartbeat from core client for 30 sec - exiting 03:35:22 (3632): No heartbeat from core client for 30 sec - exiting 03:35:23 (3632): No heartbeat from core client for 30 sec - exiting 03:35:24 (3632): No heartbeat from core client for 30 sec - exiting 03:35:25 (3632): No heartbeat from core client for 30 sec - exiting 03:35:26 (3632): No heartbeat from core client for 30 sec - exiting 03:35:27 (3632): No heartbeat from core client for 30 sec - exiting 03:35:28 (3632): No heartbeat from core client for 30 sec - exiting 03:35:30 (3632): No heartbeat from core client for 30 sec - exiting 03:35:31 (3632): No heartbeat from core client for 30 sec - exiting 03:35:32 (3632): No heartbeat from core client for 30 sec - exiting 03:35:33 (3632): No heartbeat from core client for 30 sec - exiting 03:35:34 (3632): No heartbeat from core client for 30 sec - exiting 03:35:35 (3632): No heartbeat from core client for 30 sec - exiting 03:35:36 (3632): No heartbeat from core client for 30 sec - exiting 03:35:37 (3632): No heartbeat from core client for 30 sec - exiting 03:35:38 (3632): No heartbeat from core client for 30 sec - exiting 03:35:39 (3632): No heartbeat from core client for 30 sec - exiting 03:35:40 (3632): No heartbeat from core client for 30 sec - exiting 03:35:42 (3632): No heartbeat from core client for 30 sec - exiting 03:35:43 (3632): No heartbeat from core client for 30 sec - exiting 03:35:44 (3632): No heartbeat from core client for 30 sec - exiting 03:35:45 (3632): No heartbeat from core client for 30 sec - exiting 03:35:46 (3632): No heartbeat from core client for 30 sec - exiting 03:35:47 (3632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:35:48 (3632): No heartbeat from core client for 30 sec - exiting 03:35:49 (3632): No heartbeat from core client for 30 sec - exiting 03:35:50 (3632): No heartbeat from core client for 30 sec - exiting 03:35:51 (3632): No heartbeat from core client for 30 sec - exiting 03:35:52 (3632): No heartbeat from core client for 30 sec - exiting 03:35:54 (3632): No heartbeat from core client for 30 sec - exiting 03:35:55 (3632): No heartbeat from core client for 30 sec - exiting 03:35:56 (3632): No heartbeat from core client for 30 sec - exiting 03:35:57 (3632): No heartbeat from core client for 30 sec - exiting 03:35:58 (3632): No heartbeat from core client for 30 sec - exiting 03:38:00 (4508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:19 (3780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:44:09 (3100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:33 (2212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:00:32 (4856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:00:54 (4856): No heartbeat from core client for 30 sec - exiting 02:00:55 (4856): No heartbeat from core client for 30 sec - exiting 02:00:57 (4856): No heartbeat from core client for 30 sec - exiting 02:00:58 (4856): No heartbeat from core client for 30 sec - exiting 02:00:59 (4856): No heartbeat from core client for 30 sec - exiting 02:01:00 (4856): No heartbeat from core client for 30 sec - exiting 02:01:01 (4856): No heartbeat from core client for 30 sec - exiting 02:01:02 (4856): No heartbeat from core client for 30 sec - exiting 02:01:03 (4856): No heartbeat from core client for 30 sec - exiting 02:01:04 (4856): No heartbeat from core client for 30 sec - exiting 02:01:05 (4856): No heartbeat from core client for 30 sec - exiting 02:01:06 (4856): No heartbeat from core client for 30 sec - exiting 02:01:07 (4856): No heartbeat from core client for 30 sec - exiting 02:01:09 (4856): No heartbeat from core client for 30 sec - exiting 02:01:10 (4856): No heartbeat from core client for 30 sec - exiting 02:01:11 (4856): No heartbeat from core client for 30 sec - exiting 02:01:12 (4856): No heartbeat from core client for 30 sec - exiting 02:01:13 (4856): No heartbeat from core client for 30 sec - exiting 02:01:14 (4856): No heartbeat from core client for 30 sec - exiting 02:01:15 (4856): No heartbeat from core client for 30 sec - exiting 02:01:16 (4856): No heartbeat from core client for 30 sec - exiting 02:01:17 (4856): No heartbeat from core client for 30 sec - exiting 02:01:18 (4856): No heartbeat from core client for 30 sec - exiting 02:01:52 (4856): No heartbeat from core client for 30 sec - exiting 02:01:53 (4856): No heartbeat from core client for 30 sec - exiting 16:16:53 (6440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:17:13 (6440): No heartbeat from core client for 30 sec - exiting 16:17:14 (6440): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 22:46:28 (4340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:27 (6368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:19:10 (5408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:22:51 (5080): No heartbeat from core client for 30 sec - exiting 02:22:52 (5080): No heartbeat from core client for 30 sec - exiting 02:22:53 (5080): No heartbeat from core client for 30 sec - exiting 02:22:54 (5080): No heartbeat from core client for 30 sec - exiting 02:22:55 (5080): No heartbeat from core client for 30 sec - exiting 02:22:56 (5080): No heartbeat from core client for 30 sec - exiting 02:22:57 (5080): No heartbeat from core client for 30 sec - exiting 02:22:58 (5080): No heartbeat from core client for 30 sec - exiting 02:22:59 (5080): No heartbeat from core client for 30 sec - exiting 02:23:00 (5080): No heartbeat from core client for 30 sec - exiting 02:23:02 (5080): No heartbeat from core client for 30 sec - exiting 02:23:03 (5080): No heartbeat from core client for 30 sec - exiting 02:23:04 (5080): No heartbeat from core client for 30 sec - exiting 02:23:05 (5080): No heartbeat from core client for 30 sec - exiting 02:23:06 (5080): No heartbeat from core client for 30 sec - exiting 02:23:07 (5080): No heartbeat from core client for 30 sec - exiting 02:23:08 (5080): No heartbeat from core client for 30 sec - exiting 02:23:09 (5080): No heartbeat from core client for 30 sec - exiting 02:23:10 (5080): No heartbeat from core client for 30 sec - exiting 02:23:11 (5080): No heartbeat from core client for 30 sec - exiting 02:23:13 (5080): No heartbeat from core client for 30 sec - exiting 02:23:14 (5080): No heartbeat from core client for 30 sec - exiting 02:23:15 (5080): No heartbeat from core client for 30 sec - exiting 02:23:16 (5080): No heartbeat from core client for 30 sec - exiting 02:23:17 (5080): No heartbeat from core client for 30 sec - exiting 02:23:18 (5080): No heartbeat from core client for 30 sec - exiting 02:23:19 (5080): No heartbeat from core client for 30 sec - exiting 02:23:20 (5080): No heartbeat from core client for 30 sec - exiting 02:23:21 (5080): No heartbeat from core client for 30 sec - exiting 02:23:22 (5080): No heartbeat from core client for 30 sec - exiting 02:23:23 (5080): No heartbeat from core client for 30 sec - exiting 02:23:25 (5080): No heartbeat from core client for 30 sec - exiting 02:23:26 (5080): No heartbeat from core client for 30 sec - exiting 02:23:27 (5080): No heartbeat from core client for 30 sec - exiting 02:23:28 (5080): No heartbeat from core client for 30 sec - exiting 02:23:29 (5080): No heartbeat from core client for 30 sec - exiting 02:23:30 (5080): No heartbeat from core client for 30 sec - exiting 02:23:31 (5080): No heartbeat from core client for 30 sec - exiting 02:23:32 (5080): No heartbeat from core client for 30 sec - exiting 02:23:33 (5080): No heartbeat from core client for 30 sec - exiting 02:23:34 (5080): No heartbeat from core client for 30 sec - exiting 02:23:35 (5080): No heartbeat from core client for 30 sec - exiting 02:23:37 (5080): No heartbeat from core client for 30 sec - exiting 02:23:38 (5080): No heartbeat from core client for 30 sec - exiting 02:23:39 (5080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:54:49 (5364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:56:40 (5364): No heartbeat from core client for 30 sec - exiting 01:56:41 (5364): No heartbeat from core client for 30 sec - exiting 01:56:42 (5364): No heartbeat from core client for 30 sec - exiting 01:56:43 (5364): No heartbeat from core client for 30 sec - exiting 01:56:44 (5364): No heartbeat from core client for 30 sec - exiting 01:56:45 (5364): No heartbeat from core client for 30 sec - exiting 01:56:46 (5364): No heartbeat from core client for 30 sec - exiting 01:56:48 (5364): No heartbeat from core client for 30 sec - exiting 01:56:49 (5364): No heartbeat from core client for 30 sec - exiting 01:56:50 (5364): No heartbeat from core client for 30 sec - exiting 01:56:51 (5364): No heartbeat from core client for 30 sec - exiting 01:56:52 (5364): No heartbeat from core client for 30 sec - exiting 01:56:53 (5364): No heartbeat from core client for 30 sec - exiting 01:56:54 (5364): No heartbeat from core client for 30 sec - exiting 01:56:55 (5364): No heartbeat from core client for 30 sec - exiting 01:56:56 (5364): No heartbeat from core client for 30 sec - exiting 01:56:57 (5364): No heartbeat from core client for 30 sec - exiting 01:56:59 (5364): No heartbeat from core client for 30 sec - exiting 02:17:20 (4452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:21:22 (4452): No heartbeat from core client for 30 sec - exiting 02:21:23 (4452): No heartbeat from core client for 30 sec - exiting 02:21:24 (4452): No heartbeat from core client for 30 sec - exiting 02:21:25 (4452): No heartbeat from core client for 30 sec - exiting 02:21:26 (4452): No heartbeat from core client for 30 sec - exiting 02:21:27 (4452): No heartbeat from core client for 30 sec - exiting 02:21:28 (4452): No heartbeat from core client for 30 sec - exiting 02:21:30 (4452): No heartbeat from core client for 30 sec - exiting 02:21:31 (4452): No heartbeat from core client for 30 sec - exiting 02:21:32 (4452): No heartbeat from core client for 30 sec - exiting 02:21:33 (4452): No heartbeat from core client for 30 sec - exiting 01:58:49 (1504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:55 (1504): No heartbeat from core client for 30 sec - exiting 01:58:56 (1504): No heartbeat from core client for 30 sec - exiting 02:16:18 (5616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:19:31 (5616): No heartbeat from core client for 30 sec - exiting 02:19:32 (5616): No heartbeat from core client for 30 sec - exiting 02:19:33 (5616): No heartbeat from core client for 30 sec - exiting 02:19:34 (5616): No heartbeat from core client for 30 sec - exiting 02:19:35 (5616): No heartbeat from core client for 30 sec - exiting 02:19:36 (5616): No heartbeat from core client for 30 sec - exiting 02:19:37 (5616): No heartbeat from core client for 30 sec - exiting 02:19:38 (5616): No heartbeat from core client for 30 sec - exiting 02:19:39 (5616): No heartbeat from core client for 30 sec - exiting 02:19:40 (5616): No heartbeat from core client for 30 sec - exiting 02:23:38 (1744): No heartbeat from core client for 30 sec - exiting 02:23:39 (1744): No heartbeat from core client for 30 sec - exiting 02:23:40 (1744): No heartbeat from core client for 30 sec - exiting 02:23:41 (1744): No heartbeat from core client for 30 sec - exiting 02:23:42 (1744): No heartbeat from core client for 30 sec - exiting 02:23:44 (1744): No heartbeat from core client for 30 sec - exiting 02:23:45 (1744): No heartbeat from core client for 30 sec - exiting 02:23:46 (1744): No heartbeat from core client for 30 sec - exiting 02:23:47 (1744): No heartbeat from core client for 30 sec - exiting 02:23:48 (1744): No heartbeat from core client for 30 sec - exiting 02:23:49 (1744): No heartbeat from core client for 30 sec - exiting 02:23:50 (1744): No heartbeat from core client for 30 sec - exiting 02:23:51 (1744): No heartbeat from core client for 30 sec - exiting 02:23:52 (1744): No heartbeat from core client for 30 sec - exiting 02:23:53 (1744): No heartbeat from core client for 30 sec - exiting 02:23:55 (1744): No heartbeat from core client for 30 sec - exiting 02:23:56 (1744): No heartbeat from core client for 30 sec - exiting 02:23:57 (1744): No heartbeat from core client for 30 sec - exiting 02:23:58 (1744): No heartbeat from core client for 30 sec - exiting 02:23:59 (1744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:35:04 (1616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:35:34 (1616): No heartbeat from core client for 30 sec - exiting 05:35:35 (1616): No heartbeat from core client for 30 sec - exiting 05:35:36 (1616): No heartbeat from core client for 30 sec - exiting 05:35:37 (1616): No heartbeat from core client for 30 sec - exiting 05:35:38 (1616): No heartbeat from core client for 30 sec - exiting 05:35:39 (1616): No heartbeat from core client for 30 sec - exiting 05:35:40 (1616): No heartbeat from core client for 30 sec - exiting 05:35:41 (1616): No heartbeat from core client for 30 sec - exiting 05:35:42 (1616): No heartbeat from core client for 30 sec - exiting 05:35:43 (1616): No heartbeat from core client for 30 sec - exiting 05:35:45 (1616): No heartbeat from core client for 30 sec - exiting 05:35:46 (1616): No heartbeat from core client for 30 sec - exiting 05:35:47 (1616): No heartbeat from core client for 30 sec - exiting 05:35:48 (1616): No heartbeat from core client for 30 sec - exiting 05:35:49 (1616): No heartbeat from core client for 30 sec - exiting 05:35:50 (1616): No heartbeat from core client for 30 sec - exiting 05:35:51 (1616): No heartbeat from core client for 30 sec - exiting 05:35:52 (1616): No heartbeat from core client for 30 sec - exiting 05:35:53 (1616): No heartbeat from core client for 30 sec - exiting 05:35:54 (1616): No heartbeat from core client for 30 sec - exiting 18:02:55 (5332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:03:54 (5332): No heartbeat from core client for 30 sec - exiting 18:03:55 (5332): No heartbeat from core client for 30 sec - exiting 18:03:56 (5332): No heartbeat from core client for 30 sec - exiting 18:03:57 (5332): No heartbeat from core client for 30 sec - exiting 18:03:58 (5332): No heartbeat from core client for 30 sec - exiting 18:03:59 (5332): No heartbeat from core client for 30 sec - exiting 18:04:01 (5332): No heartbeat from core client for 30 sec - exiting 18:04:02 (5332): No heartbeat from core client for 30 sec - exiting 19:18:12 (7092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:32:10 (1968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:32:56 (3280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:34:17 (4680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:34:35 (4680): No heartbeat from core client for 30 sec - exiting 19:34:37 (4680): No heartbeat from core client for 30 sec - exiting 19:34:38 (4680): No heartbeat from core client for 30 sec - exiting 19:34:39 (4680): No heartbeat from core client for 30 sec - exiting 19:34:40 (4680): No heartbeat from core client for 30 sec - exiting 19:34:41 (4680): No heartbeat from core client for 30 sec - exiting 19:34:42 (4680): No heartbeat from core client for 30 sec - exiting 19:35:46 (5288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:37:01 (5456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:38:33 (1744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:18:31 (3824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:22:23 (4708): No heartbeat from core client for 30 sec - exiting 20:22:28 (4708): No heartbeat from core client for 30 sec - exiting 20:22:30 (4708): No heartbeat from core client for 30 sec - exiting 20:22:31 (4708): No heartbeat from core client for 30 sec - exiting 20:22:32 (4708): No heartbeat from core client for 30 sec - exiting 20:22:33 (4708): No heartbeat from core client for 30 sec - exiting 20:22:34 (4708): No heartbeat from core client for 30 sec - exiting 20:22:35 (4708): No heartbeat from core client for 30 sec - exiting 20:22:36 (4708): No heartbeat from core client for 30 sec - exiting 20:22:37 (4708): No heartbeat from core client for 30 sec - exiting 20:22:38 (4708): No heartbeat from core client for 30 sec - exiting 20:22:39 (4708): No heartbeat from core client for 30 sec - exiting 20:22:41 (4708): No heartbeat from core client for 30 sec - exiting 20:22:42 (4708): No heartbeat from core client for 30 sec - exiting 20:22:43 (4708): No heartbeat from core client for 30 sec - exiting 20:22:44 (4708): No heartbeat from core client for 30 sec - exiting 20:22:45 (4708): No heartbeat from core client for 30 sec - exiting 20:22:46 (4708): No heartbeat from core client for 30 sec - exiting 20:22:47 (4708): No heartbeat from core client for 30 sec - exiting 20:22:48 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:32 (3592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:37 (3592): No heartbeat from core client for 30 sec - exiting 20:28:38 (3592): No heartbeat from core client for 30 sec - exiting 20:28:39 (3592): No heartbeat from core client for 30 sec - exiting 20:29:52 (6908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:31:00 (3916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:31:04 (3916): No heartbeat from core client for 30 sec - exiting 20:31:06 (3916): No heartbeat from core client for 30 sec - exiting 20:31:07 (3916): No heartbeat from core client for 30 sec - exiting 20:31:58 (5240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:32:07 (5240): No heartbeat from core client for 30 sec - exiting 20:32:08 (5240): No heartbeat from core client for 30 sec - exiting 20:32:09 (5240): No heartbeat from core client for 30 sec - exiting 22:46:26 (2844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:03:49 (1820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:04:14 (1820): No heartbeat from core client for 30 sec - exiting 02:04:16 (1820): No heartbeat from core client for 30 sec - exiting 02:04:17 (1820): No heartbeat from core client for 30 sec - exiting 02:04:18 (1820): No heartbeat from core client for 30 sec - exiting 02:04:19 (1820): No heartbeat from core client for 30 sec - exiting 02:04:20 (1820): No heartbeat from core client for 30 sec - exiting 02:04:21 (1820): No heartbeat from core client for 30 sec - exiting 02:04:22 (1820): No heartbeat from core client for 30 sec - exiting 02:16:58 (6116): No heartbeat from core client for 30 sec - exiting 02:16:59 (6116): No heartbeat from core client for 30 sec - exiting 02:17:00 (6116): No heartbeat from core client for 30 sec - exiting 02:17:01 (6116): No heartbeat from core client for 30 sec - exiting 02:17:02 (6116): No heartbeat from core client for 30 sec - exiting 02:17:03 (6116): No heartbeat from core client for 30 sec - exiting 02:17:04 (6116): No heartbeat from core client for 30 sec - exiting 02:17:05 (6116): No heartbeat from core client for 30 sec - exiting 02:17:06 (6116): No heartbeat from core client for 30 sec - exiting 02:17:07 (6116): No heartbeat from core client for 30 sec - exiting 02:17:09 (6116): No heartbeat from core client for 30 sec - exiting 02:17:10 (6116): No heartbeat from core client for 30 sec - exiting 02:17:11 (6116): No heartbeat from core client for 30 sec - exiting 02:17:12 (6116): No heartbeat from core client for 30 sec - exiting 02:17:13 (6116): No heartbeat from core client for 30 sec - exiting 02:17:14 (6116): No heartbeat from core client for 30 sec - exiting 02:17:15 (6116): No heartbeat from core client for 30 sec - exiting 02:17:16 (6116): No heartbeat from core client for 30 sec - exiting 02:17:17 (6116): No heartbeat from core client for 30 sec - exiting 02:17:18 (6116): No heartbeat from core client for 30 sec - exiting 02:17:19 (6116): No heartbeat from core client for 30 sec - exiting 02:17:21 (6116): No heartbeat from core client for 30 sec - exiting 02:17:22 (6116): No heartbeat from core client for 30 sec - exiting 02:17:23 (6116): No heartbeat from core client for 30 sec - exiting 02:17:24 (6116): No heartbeat from core client for 30 sec - exiting 02:17:25 (6116): No heartbeat from core client for 30 sec - exiting 02:17:26 (6116): No heartbeat from core client for 30 sec - exiting 02:17:27 (6116): No heartbeat from core client for 30 sec - exiting 02:17:28 (6116): No heartbeat from core client for 30 sec - exiting 02:17:29 (6116): No heartbeat from core client for 30 sec - exiting 02:17:30 (6116): No heartbeat from core client for 30 sec - exiting 02:17:31 (6116): No heartbeat from core client for 30 sec - exiting 02:17:33 (6116): No heartbeat from core client for 30 sec - exiting 02:17:34 (6116): No heartbeat from core client for 30 sec - exiting 02:17:35 (6116): No heartbeat from core client for 30 sec - exiting 02:17:36 (6116): No heartbeat from core client for 30 sec - exiting 02:17:37 (6116): No heartbeat from core client for 30 sec - exiting 02:17:38 (6116): No heartbeat from core client for 30 sec - exiting 02:17:39 (6116): No heartbeat from core client for 30 sec - exiting 02:17:40 (6116): No heartbeat from core client for 30 sec - exiting 02:17:41 (6116): No heartbeat from core client for 30 sec - exiting 02:17:42 (6116): No heartbeat from core client for 30 sec - exiting 02:17:43 (6116): No heartbeat from core client for 30 sec - exiting 02:17:45 (6116): No heartbeat from core client for 30 sec - exiting 02:17:46 (6116): No heartbeat from core client for 30 sec - exiting 02:17:47 (6116): No heartbeat from core client for 30 sec - exiting 02:17:48 (6116): No heartbeat from core client for 30 sec - exiting 02:17:49 (6116): No heartbeat from core client for 30 sec - exiting 02:17:50 (6116): No heartbeat from core client for 30 sec - exiting 02:17:51 (6116): No heartbeat from core client for 30 sec - exiting 02:17:52 (6116): No heartbeat from core client for 30 sec - exiting 02:17:53 (6116): No heartbeat from core client for 30 sec - exiting 02:17:54 (6116): No heartbeat from core client for 30 sec - exiting 02:17:55 (6116): No heartbeat from core client for 30 sec - exiting 02:17:57 (6116): No heartbeat from core client for 30 sec - exiting 02:17:58 (6116): No heartbeat from core client for 30 sec - exiting 02:17:59 (6116): No heartbeat from core client for 30 sec - exiting 02:18:00 (6116): No heartbeat from core client for 30 sec - exiting 02:18:01 (6116): No heartbeat from core client for 30 sec - exiting 02:18:02 (6116): No heartbeat from core client for 30 sec - exiting 02:18:03 (6116): No heartbeat from core client for 30 sec - exiting 02:18:04 (6116): No heartbeat from core client for 30 sec - exiting 02:18:38 (6116): No heartbeat from core client for 30 sec - exiting 02:18:39 (6116): No heartbeat from core client for 30 sec - exiting 02:18:40 (6116): No heartbeat from core client for 30 sec - exiting 02:18:41 (6116): No heartbeat from core client for 30 sec - exiting 02:18:42 (6116): No heartbeat from core client for 30 sec - exiting 02:18:44 (6116): No heartbeat from core client for 30 sec - exiting 02:18:45 (6116): No heartbeat from core client for 30 sec - exiting 02:18:46 (6116): No heartbeat from core client for 30 sec - exiting 02:18:47 (6116): No heartbeat from core client for 30 sec - exiting 02:18:48 (6116): No heartbeat from core client for 30 sec - exiting 02:18:49 (6116): No heartbeat from core client for 30 sec - exiting 02:18:50 (6116): No heartbeat from core client for 30 sec - exiting 02:18:51 (6116): No heartbeat from core client for 30 sec - exiting 02:18:52 (6116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:10:28 (4756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:10:34 (4756): No heartbeat from core client for 30 sec - exiting 02:10:35 (4756): No heartbeat from core client for 30 sec - exiting 02:10:36 (4756): No heartbeat from core client for 30 sec - exiting 02:10:37 (4756): No heartbeat from core client for 30 sec - exiting 02:17:28 (3672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:21:16 (6856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:22:03 (6580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:23:32 (4524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:24:31 (3540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:25:26 (4296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:26:33 (4504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:27:32 (6056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:28:53 (944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:30:06 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:31:26 (4556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:32:42 (3704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:34:12 (6836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:34:13 (6836): No heartbeat from core client for 30 sec - exiting 22:26:25 (4612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:11:17 (1732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:11:26 (1732): No heartbeat from core client for 30 sec - exiting 02:11:27 (1732): No heartbeat from core client for 30 sec - exiting 02:11:28 (1732): No heartbeat from core client for 30 sec - exiting 02:11:29 (1732): No heartbeat from core client for 30 sec - exiting 02:11:30 (1732): No heartbeat from core client for 30 sec - exiting 02:11:31 (1732): No heartbeat from core client for 30 sec - exiting 02:11:32 (1732): No heartbeat from core client for 30 sec - exiting 02:19:28 (7096): No heartbeat from core client for 30 sec - exiting 02:19:32 (7096): No heartbeat from core client for 30 sec - exiting 02:19:33 (7096): No heartbeat from core client for 30 sec - exiting 02:19:37 (7096): No heartbeat from core client for 30 sec - exiting 02:19:38 (7096): No heartbeat from core client for 30 sec - exiting 02:19:41 (7096): No heartbeat from core client for 30 sec - exiting 02:19:42 (7096): No heartbeat from core client for 30 sec - exiting 02:19:43 (7096): No heartbeat from core client for 30 sec - exiting 02:19:44 (7096): No heartbeat from core client for 30 sec - exiting 02:19:49 (7096): No heartbeat from core client for 30 sec - exiting 02:19:50 (7096): No heartbeat from core client for 30 sec - exiting 02:19:51 (7096): No heartbeat from core client for 30 sec - exiting 02:19:53 (7096): No heartbeat from core client for 30 sec - exiting 02:19:54 (7096): No heartbeat from core client for 30 sec - exiting 02:19:55 (7096): No heartbeat from core client for 30 sec - exiting 02:19:56 (7096): No heartbeat from core client for 30 sec - exiting 02:19:57 (7096): No heartbeat from core client for 30 sec - exiting 02:19:58 (7096): No heartbeat from core client for 30 sec - exiting 02:19:59 (7096): No heartbeat from core client for 30 sec - exiting 02:20:00 (7096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:21:53 (1744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:23:05 (2680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4816, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4816, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4816, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4816, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4816, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4816, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Jun 2012 23:09:37	1122757	14788932	hadcm3n_y96l_1980_40_008000417_0	207,360	812,233	3.9170
22 Jun 2012 15:09:54	1122757	14788932	hadcm3n_y96l_1980_40_008000417_0	181,440	706,027	3.8912
21 Jun 2012 10:02:38	1122757	14788932	hadcm3n_y96l_1980_40_008000417_0	155,520	616,290	3.9628
19 Jun 2012 22:08:00	1122757	14788932	hadcm3n_y96l_1980_40_008000417_0	129,600	501,685	3.8710
18 Jun 2012 14:11:28	1122757	14788932	hadcm3n_y96l_1980_40_008000417_0	103,680	397,136	3.8304
17 Jun 2012 00:58:08	1122757	14788932	hadcm3n_y96l_1980_40_008000417_0	77,760	278,546	3.5821
15 Jun 2012 17:26:05	1122757	14788932	hadcm3n_y96l_1980_40_008000417_0	51,840	174,375	3.3637
14 Jun 2012 13:06:08	1122757	14788932	hadcm3n_y96l_1980_40_008000417_0	25,920	84,323	3.2532