Task 13783474

Name	hadcm3n_y9q0_1940_40_007615853_2
Workunit	7793983
Created	15 Dec 2011, 16:32:30 UTC
Sent	15 Dec 2011, 16:33:46 UTC
Report deadline	16 Mar 2012, 0:00:57 UTC
Received	20 Feb 2012, 10:44:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	941998
Run time	15 days 20 hours 34 min 25 sec
CPU time	15 days 20 hours 34 min 25 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	0.95 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.4.5</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 09:42:43 (5572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:48:41 (6040): No heartbeat from core client for 30 sec - exiting 05:48:42 (6040): No heartbeat from core client for 30 sec - exiting 05:48:43 (6040): No heartbeat from core client for 30 sec - exiting 05:48:44 (6040): No heartbeat from core client for 30 sec - exiting 05:48:45 (6040): No heartbeat from core client for 30 sec - exiting 05:48:46 (6040): No heartbeat from core client for 30 sec - exiting 05:48:47 (6040): No heartbeat from core client for 30 sec - exiting 05:48:48 (6040): No heartbeat from core client for 30 sec - exiting 05:48:49 (6040): No heartbeat from core client for 30 sec - exiting 05:48:50 (6040): No heartbeat from core client for 30 sec - exiting 05:48:51 (6040): No heartbeat from core client for 30 sec - exiting 05:48:52 (6040): No heartbeat from core client for 30 sec - exiting 05:48:53 (6040): No heartbeat from core client for 30 sec - exiting 05:48:54 (6040): No heartbeat from core client for 30 sec - exiting 05:48:55 (6040): No heartbeat from core client for 30 sec - exiting 05:48:56 (6040): No heartbeat from core client for 30 sec - exiting 05:48:57 (6040): No heartbeat from core client for 30 sec - exiting 05:48:58 (6040): No heartbeat from core client for 30 sec - exiting 05:48:59 (6040): No heartbeat from core client for 30 sec - exiting 05:49:00 (6040): No heartbeat from core client for 30 sec - exiting 05:49:01 (6040): No heartbeat from core client for 30 sec - exiting 05:49:02 (6040): No heartbeat from core client for 30 sec - exiting 05:49:03 (6040): No heartbeat from core client for 30 sec - exiting 05:49:04 (6040): No heartbeat from core client for 30 sec - exiting 05:49:05 (6040): No heartbeat from core client for 30 sec - exiting 05:49:06 (6040): No heartbeat from core client for 30 sec - exiting 05:49:08 (6040): No heartbeat from core client for 30 sec - exiting 05:49:09 (6040): No heartbeat from core client for 30 sec - exiting 05:49:10 (6040): No heartbeat from core client for 30 sec - exiting 05:49:11 (6040): No heartbeat from core client for 30 sec - exiting 05:49:12 (6040): No heartbeat from core client for 30 sec - exiting 05:49:13 (6040): No heartbeat from core client for 30 sec - exiting 05:49:14 (6040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:37 (4304): No heartbeat from core client for 30 sec - exiting 18:30:38 (4304): No heartbeat from core client for 30 sec - exiting 18:30:40 (4304): No heartbeat from core client for 30 sec - exiting 18:30:41 (4304): No heartbeat from core client for 30 sec - exiting 18:30:42 (4304): No heartbeat from core client for 30 sec - exiting 18:30:43 (4304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:24 (5492): No heartbeat from core client for 30 sec - exiting 07:31:25 (5492): No heartbeat from core client for 30 sec - exiting 07:31:26 (5492): No heartbeat from core client for 30 sec - exiting 07:31:27 (5492): No heartbeat from core client for 30 sec - exiting 07:31:28 (5492): No heartbeat from core client for 30 sec - exiting 07:31:29 (5492): No heartbeat from core client for 30 sec - exiting 07:31:30 (5492): No heartbeat from core client for 30 sec - exiting 07:31:31 (5492): No heartbeat from core client for 30 sec - exiting 07:31:32 (5492): No heartbeat from core client for 30 sec - exiting 07:31:33 (5492): No heartbeat from core client for 30 sec - exiting 07:31:34 (5492): No heartbeat from core client for 30 sec - exiting 07:31:35 (5492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4924, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5360, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7224, iMonCtr=1 Model crash detected, will try to restart... 17:42:57 (5908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:29 (4860): No heartbeat from core client for 30 sec - exiting 07:37:30 (4860): No heartbeat from core client for 30 sec - exiting 07:37:31 (4860): No heartbeat from core client for 30 sec - exiting 07:37:32 (4860): No heartbeat from core client for 30 sec - exiting 07:37:33 (4860): No heartbeat from core client for 30 sec - exiting 07:37:34 (4860): No heartbeat from core client for 30 sec - exiting 07:37:35 (4860): No heartbeat from core client for 30 sec - exiting 07:37:36 (4860): No heartbeat from core client for 30 sec - exiting 07:37:37 (4860): No heartbeat from core client for 30 sec - exiting 07:37:38 (4860): No heartbeat from core client for 30 sec - exiting 07:37:39 (4860): No heartbeat from core client for 30 sec - exiting 07:37:40 (4860): No heartbeat from core client for 30 sec - exiting 07:37:41 (4860): No heartbeat from core client for 30 sec - exiting 07:37:42 (4860): No heartbeat from core client for 30 sec - exiting 07:37:43 (4860): No heartbeat from core client for 30 sec - exiting 07:37:44 (4860): No heartbeat from core client for 30 sec - exiting 07:37:45 (4860): No heartbeat from core client for 30 sec - exiting 07:37:46 (4860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:12 (5880): No heartbeat from core client for 30 sec - exiting 18:25:13 (5880): No heartbeat from core client for 30 sec - exiting 18:25:14 (5880): No heartbeat from core client for 30 sec - exiting 18:25:16 (5880): No heartbeat from core client for 30 sec - exiting 18:25:17 (5880): No heartbeat from core client for 30 sec - exiting 18:25:18 (5880): No heartbeat from core client for 30 sec - exiting 18:25:19 (5880): No heartbeat from core client for 30 sec - exiting 18:25:20 (5880): No heartbeat from core client for 30 sec - exiting 18:25:21 (5880): No heartbeat from core client for 30 sec - exiting 18:25:22 (5880): No heartbeat from core client for 30 sec - exiting 18:25:23 (5880): No heartbeat from core client for 30 sec - exiting 18:25:24 (5880): No heartbeat from core client for 30 sec - exiting 18:25:25 (5880): No heartbeat from core client for 30 sec - exiting 18:25:26 (5880): No heartbeat from core client for 30 sec - exiting 18:25:27 (5880): No heartbeat from core client for 30 sec - exiting 18:25:28 (5880): No heartbeat from core client for 30 sec - exiting 18:25:29 (5880): No heartbeat from core client for 30 sec - exiting 18:25:30 (5880): No heartbeat from core client for 30 sec - exiting 18:25:31 (5880): No heartbeat from core client for 30 sec - exiting 18:25:32 (5880): No heartbeat from core client for 30 sec - exiting 18:25:33 (5880): No heartbeat from core client for 30 sec - exiting 18:25:34 (5880): No heartbeat from core client for 30 sec - exiting 18:25:35 (5880): No heartbeat from core client for 30 sec - exiting 18:25:36 (5880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:37 (5880): No heartbeat from core client for 30 sec - exiting 18:25:38 (5880): No heartbeat from core client for 30 sec - exiting 18:25:39 (5880): No heartbeat from core client for 30 sec - exiting 18:25:40 (5880): No heartbeat from core client for 30 sec - exiting 18:25:41 (5880): No heartbeat from core client for 30 sec - exiting 18:25:42 (5880): No heartbeat from core client for 30 sec - exiting 18:25:43 (5880): No heartbeat from core client for 30 sec - exiting 18:10:30 (5076): No heartbeat from core client for 30 sec - exiting 18:10:31 (5076): No heartbeat from core client for 30 sec - exiting 18:10:32 (5076): No heartbeat from core client for 30 sec - exiting 18:10:33 (5076): No heartbeat from core client for 30 sec - exiting 18:10:34 (5076): No heartbeat from core client for 30 sec - exiting 18:10:35 (5076): No heartbeat from core client for 30 sec - exiting 18:10:36 (5076): No heartbeat from core client for 30 sec - exiting 18:10:37 (5076): No heartbeat from core client for 30 sec - exiting 18:10:38 (5076): No heartbeat from core client for 30 sec - exiting 18:10:39 (5076): No heartbeat from core client for 30 sec - exiting 18:10:40 (5076): No heartbeat from core client for 30 sec - exiting 18:10:41 (5076): No heartbeat from core client for 30 sec - exiting 18:10:42 (5076): No heartbeat from core client for 30 sec - exiting 18:10:43 (5076): No heartbeat from core client for 30 sec - exiting 18:10:44 (5076): No heartbeat from core client for 30 sec - exiting 18:10:45 (5076): No heartbeat from core client for 30 sec - exiting 18:10:46 (5076): No heartbeat from core client for 30 sec - exiting 18:10:47 (5076): No heartbeat from core client for 30 sec - exiting 18:10:48 (5076): No heartbeat from core client for 30 sec - exiting 18:10:49 (5076): No heartbeat from core client for 30 sec - exiting 18:10:50 (5076): No heartbeat from core client for 30 sec - exiting 18:10:51 (5076): No heartbeat from core client for 30 sec - exiting 18:10:52 (5076): No heartbeat from core client for 30 sec - exiting 18:10:54 (5076): No heartbeat from core client for 30 sec - exiting 18:10:55 (5076): No heartbeat from core client for 30 sec - exiting 18:10:56 (5076): No heartbeat from core client for 30 sec - exiting 18:10:57 (5076): No heartbeat from core client for 30 sec - exiting 18:10:58 (5076): No heartbeat from core client for 30 sec - exiting 18:10:59 (5076): No heartbeat from core client for 30 sec - exiting 18:11:00 (5076): No heartbeat from core client for 30 sec - exiting 18:11:01 (5076): No heartbeat from core client for 30 sec - exiting 18:11:02 (5076): No heartbeat from core client for 30 sec - exiting 18:11:03 (5076): No heartbeat from core client for 30 sec - exiting 18:11:04 (5076): No heartbeat from core client for 30 sec - exiting 18:11:05 (5076): No heartbeat from core client for 30 sec - exiting 18:11:06 (5076): No heartbeat from core client for 30 sec - exiting 18:11:07 (5076): No heartbeat from core client for 30 sec - exiting 18:11:08 (5076): No heartbeat from core client for 30 sec - exiting 18:11:09 (5076): No heartbeat from core client for 30 sec - exiting 18:11:10 (5076): No heartbeat from core client for 30 sec - exiting 18:11:11 (5076): No heartbeat from core client for 30 sec - exiting 18:11:12 (5076): No heartbeat from core client for 30 sec - exiting 18:11:13 (5076): No heartbeat from core client for 30 sec - exiting 18:11:14 (5076): No heartbeat from core client for 30 sec - exiting 18:11:15 (5076): No heartbeat from core client for 30 sec - exiting 18:11:16 (5076): No heartbeat from core client for 30 sec - exiting 18:11:17 (5076): No heartbeat from core client for 30 sec - exiting 18:11:18 (5076): No heartbeat from core client for 30 sec - exiting 18:11:19 (5076): No heartbeat from core client for 30 sec - exiting 18:11:20 (5076): No heartbeat from core client for 30 sec - exiting 18:11:21 (5076): No heartbeat from core client for 30 sec - exiting 18:11:22 (5076): No heartbeat from core client for 30 sec - exiting 18:11:23 (5076): No heartbeat from core client for 30 sec - exiting 18:11:24 (5076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:25 (5076): No heartbeat from core client for 30 sec - exiting 18:31:09 (3620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1192, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5728, iMonCtr=1 Model crash detected, will try to restart... 18:19:16 (5672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:06:33 (5712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5784, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6132, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5716, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5384, iMonCtr=1 Model crash detected, will try to restart... 18:17:31 (5924): No heartbeat from core client for 30 sec - exiting 18:17:33 (5924): No heartbeat from core client for 30 sec - exiting 18:17:34 (5924): No heartbeat from core client for 30 sec - exiting 18:17:35 (5924): No heartbeat from core client for 30 sec - exiting 18:17:36 (5924): No heartbeat from core client for 30 sec - exiting 18:17:38 (5924): No heartbeat from core client for 30 sec - exiting 18:17:39 (5924): No heartbeat from core client for 30 sec - exiting 18:17:40 (5924): No heartbeat from core client for 30 sec - exiting 18:17:41 (5924): No heartbeat from core client for 30 sec - exiting 18:17:42 (5924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:33:12 (4136): No heartbeat from core client for 30 sec - exiting 18:33:14 (4136): No heartbeat from core client for 30 sec - exiting 18:33:15 (4136): No heartbeat from core client for 30 sec - exiting 18:33:16 (4136): No heartbeat from core client for 30 sec - exiting 18:33:17 (4136): No heartbeat from core client for 30 sec - exiting 18:33:18 (4136): No heartbeat from core client for 30 sec - exiting 18:33:19 (4136): No heartbeat from core client for 30 sec - exiting 18:33:20 (4136): No heartbeat from core client for 30 sec - exiting 18:33:21 (4136): No heartbeat from core client for 30 sec - exiting 18:33:22 (4136): No heartbeat from core client for 30 sec - exiting 18:33:23 (4136): No heartbeat from core client for 30 sec - exiting 18:33:24 (4136): No heartbeat from core client for 30 sec - exiting 18:33:25 (4136): No heartbeat from core client for 30 sec - exiting 18:33:26 (4136): No heartbeat from core client for 30 sec - exiting 18:33:27 (4136): No heartbeat from core client for 30 sec - exiting 18:33:28 (4136): No heartbeat from core client for 30 sec - exiting 18:33:29 (4136): No heartbeat from core client for 30 sec - exiting 18:33:30 (4136): No heartbeat from core client for 30 sec - exiting 18:33:31 (4136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:33:32 (4136): No heartbeat from core client for 30 sec - exiting 18:33:33 (4136): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5252, iMonCtr=1 Model crash detected, will try to restart... 22:55:44 (5448): No heartbeat from core client for 30 sec - exiting 22:55:45 (5448): No heartbeat from core client for 30 sec - exiting 22:55:46 (5448): No heartbeat from core client for 30 sec - exiting 22:55:47 (5448): No heartbeat from core client for 30 sec - exiting 22:55:48 (5448): No heartbeat from core client for 30 sec - exiting 22:55:49 (5448): No heartbeat from core client for 30 sec - exiting 22:55:50 (5448): No heartbeat from core client for 30 sec - exiting 22:55:51 (5448): No heartbeat from core client for 30 sec - exiting 22:55:52 (5448): No heartbeat from core client for 30 sec - exiting 22:55:53 (5448): No heartbeat from core client for 30 sec - exiting 22:55:54 (5448): No heartbeat from core client for 30 sec - exiting 22:55:55 (5448): No heartbeat from core client for 30 sec - exiting 22:55:56 (5448): No heartbeat from core client for 30 sec - exiting 22:55:57 (5448): No heartbeat from core client for 30 sec - exiting 22:55:58 (5448): No heartbeat from core client for 30 sec - exiting 22:55:59 (5448): No heartbeat from core client for 30 sec - exiting 22:56:01 (5448): No heartbeat from core client for 30 sec - exiting 22:56:02 (5448): No heartbeat from core client for 30 sec - exiting 22:56:08 (5448): No heartbeat from core client for 30 sec - exiting 22:56:09 (5448): No heartbeat from core client for 30 sec - exiting 22:56:10 (5448): No heartbeat from core client for 30 sec - exiting 22:56:11 (5448): No heartbeat from core client for 30 sec - exiting 22:56:12 (5448): No heartbeat from core client for 30 sec - exiting 22:56:13 (5448): No heartbeat from core client for 30 sec - exiting 22:56:14 (5448): No heartbeat from core client for 30 sec - exiting 22:56:15 (5448): No heartbeat from core client for 30 sec - exiting 22:56:16 (5448): No heartbeat from core client for 30 sec - exiting 22:56:17 (5448): No heartbeat from core client for 30 sec - exiting 22:56:18 (5448): No heartbeat from core client for 30 sec - exiting 22:56:19 (5448): No heartbeat from core client for 30 sec - exiting 22:56:20 (5448): No heartbeat from core client for 30 sec - exiting 22:56:21 (5448): No heartbeat from core client for 30 sec - exiting 22:56:22 (5448): No heartbeat from core client for 30 sec - exiting 22:56:23 (5448): No heartbeat from core client for 30 sec - exiting 22:56:25 (5448): No heartbeat from core client for 30 sec - exiting 22:56:26 (5448): No heartbeat from core client for 30 sec - exiting 22:56:27 (5448): No heartbeat from core client for 30 sec - exiting 22:56:28 (5448): No heartbeat from core client for 30 sec - exiting 22:56:29 (5448): No heartbeat from core client for 30 sec - exiting 22:56:30 (5448): No heartbeat from core client for 30 sec - exiting 22:56:31 (5448): No heartbeat from core client for 30 sec - exiting 22:56:32 (5448): No heartbeat from core client for 30 sec - exiting 22:56:33 (5448): No heartbeat from core client for 30 sec - exiting 22:56:34 (5448): No heartbeat from core client for 30 sec - exiting 22:56:35 (5448): No heartbeat from core client for 30 sec - exiting 22:56:36 (5448): No heartbeat from core client for 30 sec - exiting 22:56:37 (5448): No heartbeat from core client for 30 sec - exiting 22:56:38 (5448): No heartbeat from core client for 30 sec - exiting 22:56:39 (5448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5368, iMonCtr=1 Model crash detected, will try to restart... 05:32:51 (5052): No heartbeat from core client for 30 sec - exiting 05:32:52 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 05:48:18 (5728): No heartbeat from core client for 30 sec - exiting 05:48:19 (5728): No heartbeat from core client for 30 sec - exiting 05:48:20 (5728): No heartbeat from core client for 30 sec - exiting 05:48:21 (5728): No heartbeat from core client for 30 sec - exiting 05:48:22 (5728): No heartbeat from core client for 30 sec - exiting 05:48:23 (5728): No heartbeat from core client for 30 sec - exiting 05:48:24 (5728): No heartbeat from core client for 30 sec - exiting 05:48:25 (5728): No heartbeat from core client for 30 sec - exiting 05:48:26 (5728): No heartbeat from core client for 30 sec - exiting 05:48:27 (5728): No heartbeat from core client for 30 sec - exiting 05:48:28 (5728): No heartbeat from core client for 30 sec - exiting 05:48:29 (5728): No heartbeat from core client for 30 sec - exiting 05:48:30 (5728): No heartbeat from core client for 30 sec - exiting 05:48:31 (5728): No heartbeat from core client for 30 sec - exiting 05:48:32 (5728): No heartbeat from core client for 30 sec - exiting 05:48:33 (5728): No heartbeat from core client for 30 sec - exiting 05:48:34 (5728): No heartbeat from core client for 30 sec - exiting 05:48:35 (5728): No heartbeat from core client for 30 sec - exiting 05:48:36 (5728): No heartbeat from core client for 30 sec - exiting 05:48:37 (5728): No heartbeat from core client for 30 sec - exiting 05:48:38 (5728): No heartbeat from core client for 30 sec - exiting 05:48:39 (5728): No heartbeat from core client for 30 sec - exiting 05:48:40 (5728): No heartbeat from core client for 30 sec - exiting 05:48:41 (5728): No heartbeat from core client for 30 sec - exiting 05:48:42 (5728): No heartbeat from core client for 30 sec - exiting 05:48:43 (5728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:39:05 (2480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:22:24 (4572): No heartbeat from core client for 30 sec - exiting 18:22:25 (4572): No heartbeat from core client for 30 sec - exiting 18:22:26 (4572): No heartbeat from core client for 30 sec - exiting 18:22:27 (4572): No heartbeat from core client for 30 sec - exiting 18:22:28 (4572): No heartbeat from core client for 30 sec - exiting 18:22:29 (4572): No heartbeat from core client for 30 sec - exiting 18:22:30 (4572): No heartbeat from core client for 30 sec - exiting 18:22:31 (4572): No heartbeat from core client for 30 sec - exiting 18:22:32 (4572): No heartbeat from core client for 30 sec - exiting 18:22:33 (4572): No heartbeat from core client for 30 sec - exiting 18:22:34 (4572): No heartbeat from core client for 30 sec - exiting 18:22:35 (4572): No heartbeat from core client for 30 sec - exiting 18:22:36 (4572): No heartbeat from core client for 30 sec - exiting 18:22:37 (4572): No heartbeat from core client for 30 sec - exiting 18:22:38 (4572): No heartbeat from core client for 30 sec - exiting 18:22:39 (4572): No heartbeat from core client for 30 sec - exiting 18:22:41 (4572): No heartbeat from core client for 30 sec - exiting 18:22:42 (4572): No heartbeat from core client for 30 sec - exiting 18:22:43 (4572): No heartbeat from core client for 30 sec - exiting 18:22:44 (4572): No heartbeat from core client for 30 sec - exiting 18:22:45 (4572): No heartbeat from core client for 30 sec - exiting 18:22:46 (4572): No heartbeat from core client for 30 sec - exiting 18:22:47 (4572): No heartbeat from core client for 30 sec - exiting 18:22:48 (4572): No heartbeat from core client for 30 sec - exiting 18:22:49 (4572): No heartbeat from core client for 30 sec - exiting 18:22:50 (4572): No heartbeat from core client for 30 sec - exiting 18:22:51 (4572): No heartbeat from core client for 30 sec - exiting 18:22:52 (4572): No heartbeat from core client for 30 sec - exiting 18:22:53 (4572): No heartbeat from core client for 30 sec - exiting 18:22:54 (4572): No heartbeat from core client for 30 sec - exiting 18:22:56 (4572): No heartbeat from core client for 30 sec - exiting 18:22:57 (4572): No heartbeat from core client for 30 sec - exiting 18:22:58 (4572): No heartbeat from core client for 30 sec - exiting 18:22:59 (4572): No heartbeat from core client for 30 sec - exiting 18:23:00 (4572): No heartbeat from core client for 30 sec - exiting 18:23:01 (4572): No heartbeat from core client for 30 sec - exiting 18:23:02 (4572): No heartbeat from core client for 30 sec - exiting 18:23:03 (4572): No heartbeat from core client for 30 sec - exiting 18:23:04 (4572): No heartbeat from core client for 30 sec - exiting 18:23:05 (4572): No heartbeat from core client for 30 sec - exiting 18:23:06 (4572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:44:23 (3812): No heartbeat from core client for 30 sec - exiting 22:44:25 (3812): No heartbeat from core client for 30 sec - exiting 22:44:26 (3812): No heartbeat from core client for 30 sec - exiting 22:44:27 (3812): No heartbeat from core client for 30 sec - exiting 22:44:28 (3812): No heartbeat from core client for 30 sec - exiting 22:44:29 (3812): No heartbeat from core client for 30 sec - exiting 22:44:30 (3812): No heartbeat from core client for 30 sec - exiting 22:44:31 (3812): No heartbeat from core client for 30 sec - exiting 22:44:32 (3812): No heartbeat from core client for 30 sec - exiting 22:44:33 (3812): No heartbeat from core client for 30 sec - exiting 22:44:34 (3812): No heartbeat from core client for 30 sec - exiting 22:44:35 (3812): No heartbeat from core client for 30 sec - exiting 22:44:36 (3812): No heartbeat from core client for 30 sec - exiting 22:44:37 (3812): No heartbeat from core client for 30 sec - exiting 22:44:38 (3812): No heartbeat from core client for 30 sec - exiting 22:44:39 (3812): No heartbeat from core client for 30 sec - exiting 22:44:40 (3812): No heartbeat from core client for 30 sec - exiting 22:44:41 (3812): No heartbeat from core client for 30 sec - exiting 22:44:42 (3812): No heartbeat from core client for 30 sec - exiting 22:44:43 (3812): No heartbeat from core client for 30 sec - exiting 22:44:44 (3812): No heartbeat from core client for 30 sec - exiting 22:44:45 (3812): No heartbeat from core client for 30 sec - exiting 22:44:46 (3812): No heartbeat from core client for 30 sec - exiting 22:44:49 (3812): No heartbeat from core client for 30 sec - exiting 22:44:50 (3812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4712, iMonCtr=1 Model crash detected, will try to restart... 18:19:43 (4200): No heartbeat from core client for 30 sec - exiting 18:19:44 (4200): No heartbeat from core client for 30 sec - exiting 18:19:45 (4200): No heartbeat from core client for 30 sec - exiting 18:19:46 (4200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:33:47 (5160): No heartbeat from core client for 30 sec - exiting 04:33:48 (5160): No heartbeat from core client for 30 sec - exiting 04:33:49 (5160): No heartbeat from core client for 30 sec - exiting 04:33:51 (5160): No heartbeat from core client for 30 sec - exiting 04:33:52 (5160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:01:47 (5600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:17:05 (6020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5808, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2448, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4328, iMonCtr=1 Model crash detected, will try to restart... 08:28:54 (888): No heartbeat from core client for 30 sec - exiting 08:28:55 (888): No heartbeat from core client for 30 sec - exiting 08:28:56 (888): No heartbeat from core client for 30 sec - exiting 08:28:57 (888): No heartbeat from core client for 30 sec - exiting 08:28:59 (888): No heartbeat from core client for 30 sec - exiting 08:29:00 (888): No heartbeat from core client for 30 sec - exiting 08:29:01 (888): No heartbeat from core client for 30 sec - exiting 08:29:02 (888): No heartbeat from core client for 30 sec - exiting 08:29:03 (888): No heartbeat from core client for 30 sec - exiting 08:29:05 (888): No heartbeat from core client for 30 sec - exiting 08:29:06 (888): No heartbeat from core client for 30 sec - exiting 08:29:08 (888): No heartbeat from core client for 30 sec - exiting 08:29:10 (888): No heartbeat from core client for 30 sec - exiting 08:29:12 (888): No heartbeat from core client for 30 sec - exiting 08:29:16 (888): No heartbeat from core client for 30 sec - exiting 08:29:17 (888): No heartbeat from core client for 30 sec - exiting 08:29:18 (888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:29:19 (888): No heartbeat from core client for 30 sec - exiting 08:29:20 (888): No heartbeat from core client for 30 sec - exiting 08:29:21 (888): No heartbeat from core client for 30 sec - exiting 08:29:22 (888): No heartbeat from core client for 30 sec - exiting 08:29:23 (888): No heartbeat from core client for 30 sec - exiting 08:29:24 (888): No heartbeat from core client for 30 sec - exiting 07:32:52 (2512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:53:01 (3928): No heartbeat from core client for 30 sec - exiting 16:53:02 (3928): No heartbeat from core client for 30 sec - exiting 16:53:03 (3928): No heartbeat from core client for 30 sec - exiting 16:53:04 (3928): No heartbeat from core client for 30 sec - exiting 16:53:05 (3928): No heartbeat from core client for 30 sec - exiting 16:53:06 (3928): No heartbeat from core client for 30 sec - exiting 16:53:07 (3928): No heartbeat from core client for 30 sec - exiting 16:53:08 (3928): No heartbeat from core client for 30 sec - exiting 16:53:09 (3928): No heartbeat from core client for 30 sec - exiting 16:53:10 (3928): No heartbeat from core client for 30 sec - exiting 16:53:11 (3928): No heartbeat from core client for 30 sec - exiting 16:53:12 (3928): No heartbeat from core client for 30 sec - exiting 16:53:13 (3928): No heartbeat from core client for 30 sec - exiting 16:53:14 (3928): No heartbeat from core client for 30 sec - exiting 16:53:15 (3928): No heartbeat from core client for 30 sec - exiting 16:53:16 (3928): No heartbeat from core client for 30 sec - exiting 16:53:17 (3928): No heartbeat from core client for 30 sec - exiting 16:53:18 (3928): No heartbeat from core client for 30 sec - exiting 16:53:19 (3928): No heartbeat from core client for 30 sec - exiting 16:53:20 (3928): No heartbeat from core client for 30 sec - exiting 16:53:21 (3928): No heartbeat from core client for 30 sec - exiting 16:53:22 (3928): No heartbeat from core client for 30 sec - exiting 16:53:23 (3928): No heartbeat from core client for 30 sec - exiting 16:53:24 (3928): No heartbeat from core client for 30 sec - exiting 16:53:25 (3928): No heartbeat from core client for 30 sec - exiting 16:53:26 (3928): No heartbeat from core client for 30 sec - exiting 16:53:27 (3928): No heartbeat from core client for 30 sec - exiting 16:53:28 (3928): No heartbeat from core client for 30 sec - exiting 16:53:29 (3928): No heartbeat from core client for 30 sec - exiting 16:53:30 (3928): No heartbeat from core client for 30 sec - exiting 16:53:31 (3928): No heartbeat from core client for 30 sec - exiting 16:53:32 (3928): No heartbeat from core client for 30 sec - exiting 16:53:33 (3928): No heartbeat from core client for 30 sec - exiting 16:53:34 (3928): No heartbeat from core client for 30 sec - exiting 16:53:35 (3928): No heartbeat from core client for 30 sec - exiting 16:53:37 (3928): No heartbeat from core client for 30 sec - exiting 16:53:38 (3928): No heartbeat from core client for 30 sec - exiting 16:53:39 (3928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:53:40 (3928): No heartbeat from core client for 30 sec - exiting 16:53:41 (3928): No heartbeat from core client for 30 sec - exiting 16:53:42 (3928): No heartbeat from core client for 30 sec - exiting 16:53:43 (3928): No heartbeat from core client for 30 sec - exiting 16:53:44 (3928): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4008, iMonCtr=1 Model crash detected, will try to restart... 06:28:27 (492): No heartbeat from core client for 30 sec - exiting 06:28:28 (492): No heartbeat from core client for 30 sec - exiting 06:28:29 (492): No heartbeat from core client for 30 sec - exiting 06:28:30 (492): No heartbeat from core client for 30 sec - exiting 06:28:31 (492): No heartbeat from core client for 30 sec - exiting 06:28:32 (492): No heartbeat from core client for 30 sec - exiting 06:28:33 (492): No heartbeat from core client for 30 sec - exiting 06:28:34 (492): No heartbeat from core client for 30 sec - exiting 06:28:35 (492): No heartbeat from core client for 30 sec - exiting 06:28:36 (492): No heartbeat from core client for 30 sec - exiting 06:28:37 (492): No heartbeat from core client for 30 sec - exiting 06:28:38 (492): No heartbeat from core client for 30 sec - exiting 06:28:39 (492): No heartbeat from core client for 30 sec - exiting 06:28:40 (492): No heartbeat from core client for 30 sec - exiting 06:28:41 (492): No heartbeat from core client for 30 sec - exiting 06:28:42 (492): No heartbeat from core client for 30 sec - exiting 06:28:43 (492): No heartbeat from core client for 30 sec - exiting 06:28:44 (492): No heartbeat from core client for 30 sec - exiting 06:28:45 (492): No heartbeat from core client for 30 sec - exiting 06:28:46 (492): No heartbeat from core client for 30 sec - exiting 06:28:47 (492): No heartbeat from core client for 30 sec - exiting 06:28:48 (492): No heartbeat from core client for 30 sec - exiting 06:28:49 (492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:05 (5088): No heartbeat from core client for 30 sec - exiting 21:42:06 (5088): No heartbeat from core client for 30 sec - exiting 21:42:07 (5088): No heartbeat from core client for 30 sec - exiting 21:42:08 (5088): No heartbeat from core client for 30 sec - exiting 21:42:09 (5088): No heartbeat from core client for 30 sec - exiting 21:42:10 (5088): No heartbeat from core client for 30 sec - exiting 21:42:11 (5088): No heartbeat from core client for 30 sec - exiting 21:42:12 (5088): No heartbeat from core client for 30 sec - exiting 21:42:13 (5088): No heartbeat from core client for 30 sec - exiting 21:42:14 (5088): No heartbeat from core client for 30 sec - exiting 21:42:15 (5088): No heartbeat from core client for 30 sec - exiting 21:42:16 (5088): No heartbeat from core client for 30 sec - exiting 21:42:17 (5088): No heartbeat from core client for 30 sec - exiting 21:42:18 (5088): No heartbeat from core client for 30 sec - exiting 21:42:19 (5088): No heartbeat from core client for 30 sec - exiting 21:42:20 (5088): No heartbeat from core client for 30 sec - exiting 21:42:21 (5088): No heartbeat from core client for 30 sec - exiting 21:42:22 (5088): No heartbeat from core client for 30 sec - exiting 21:42:23 (5088): No heartbeat from core client for 30 sec - exiting 21:42:24 (5088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5656, iMonCtr=1 Model crash detected, will try to restart... 07:04:05 (4248): No heartbeat from core client for 30 sec - exiting 07:04:06 (4248): No heartbeat from core client for 30 sec - exiting 07:04:07 (4248): No heartbeat from core client for 30 sec - exiting 07:04:08 (4248): No heartbeat from core client for 30 sec - exiting 07:04:09 (4248): No heartbeat from core client for 30 sec - exiting 07:04:10 (4248): No heartbeat from core client for 30 sec - exiting 07:04:11 (4248): No heartbeat from core client for 30 sec - exiting 07:04:12 (4248): No heartbeat from core client for 30 sec - exiting 07:04:13 (4248): No heartbeat from core client for 30 sec - exiting 07:04:14 (4248): No heartbeat from core client for 30 sec - exiting 07:04:15 (4248): No heartbeat from core client for 30 sec - exiting 07:04:16 (4248): No heartbeat from core client for 30 sec - exiting 07:04:17 (4248): No heartbeat from core client for 30 sec - exiting 07:04:18 (4248): No heartbeat from core client for 30 sec - exiting 07:04:19 (4248): No heartbeat from core client for 30 sec - exiting 07:04:20 (4248): No heartbeat from core client for 30 sec - exiting 07:04:21 (4248): No heartbeat from core client for 30 sec - exiting 07:04:22 (4248): No heartbeat from core client for 30 sec - exiting 07:04:23 (4248): No heartbeat from core client for 30 sec - exiting 07:04:24 (4248): No heartbeat from core client for 30 sec - exiting 07:04:25 (4248): No heartbeat from core client for 30 sec - exiting 07:04:26 (4248): No heartbeat from core client for 30 sec - exiting 07:04:27 (4248): No heartbeat from core client for 30 sec - exiting 07:04:28 (4248): No heartbeat from core client for 30 sec - exiting 07:04:29 (4248): No heartbeat from core client for 30 sec - exiting 07:04:30 (4248): No heartbeat from core client for 30 sec - exiting 07:04:31 (4248): No heartbeat from core client for 30 sec - exiting 07:04:32 (4248): No heartbeat from core client for 30 sec - exiting 07:04:33 (4248): No heartbeat from core client for 30 sec - exiting 07:04:34 (4248): No heartbeat from core client for 30 sec - exiting 07:04:35 (4248): No heartbeat from core client for 30 sec - exiting 07:04:36 (4248): No heartbeat from core client for 30 sec - exiting 07:04:37 (4248): No heartbeat from core client for 30 sec - exiting 07:04:38 (4248): No heartbeat from core client for 30 sec - exiting 07:04:39 (4248): No heartbeat from core client for 30 sec - exiting 07:04:40 (4248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:36 (5932): No heartbeat from core client for 30 sec - exiting 20:28:37 (5932): No heartbeat from core client for 30 sec - exiting 20:28:38 (5932): No heartbeat from core client for 30 sec - exiting 20:28:39 (5932): No heartbeat from core client for 30 sec - exiting 20:28:40 (5932): No heartbeat from core client for 30 sec - exiting 20:28:41 (5932): No heartbeat from core client for 30 sec - exiting 20:28:42 (5932): No heartbeat from core client for 30 sec - exiting 20:28:43 (5932): No heartbeat from core client for 30 sec - exiting 20:28:44 (5932): No heartbeat from core client for 30 sec - exiting 20:28:45 (5932): No heartbeat from core client for 30 sec - exiting 20:28:46 (5932): No heartbeat from core client for 30 sec - exiting 20:28:47 (5932): No heartbeat from core client for 30 sec - exiting 20:28:48 (5932): No heartbeat from core client for 30 sec - exiting 20:28:49 (5932): No heartbeat from core client for 30 sec - exiting 20:28:50 (5932): No heartbeat from core client for 30 sec - exiting 20:28:51 (5932): No heartbeat from core client for 30 sec - exiting 20:28:52 (5932): No heartbeat from core client for 30 sec - exiting 20:28:53 (5932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4512, iMonCtr=1 Model crash detected, will try to restart... 03:24:31 (5596): No heartbeat from core client for 30 sec - exiting 03:24:32 (5596): No heartbeat from core client for 30 sec - exiting 03:24:34 (5596): No heartbeat from core client for 30 sec - exiting 03:24:35 (5596): No heartbeat from core client for 30 sec - exiting 03:24:36 (5596): No heartbeat from core client for 30 sec - exiting 03:24:37 (5596): No heartbeat from core client for 30 sec - exiting 03:24:38 (5596): No heartbeat from core client for 30 sec - exiting 03:24:39 (5596): No heartbeat from core client for 30 sec - exiting 03:24:40 (5596): No heartbeat from core client for 30 sec - exiting 03:24:41 (5596): No heartbeat from core client for 30 sec - exiting 03:24:42 (5596): No heartbeat from core client for 30 sec - exiting 03:24:43 (5596): No heartbeat from core client for 30 sec - exiting 03:24:44 (5596): No heartbeat from core client for 30 sec - exiting 03:24:45 (5596): No heartbeat from core client for 30 sec - exiting 03:24:46 (5596): No heartbeat from core client for 30 sec - exiting 03:24:47 (5596): No heartbeat from core client for 30 sec - exiting 03:24:48 (5596): No heartbeat from core client for 30 sec - exiting 03:24:49 (5596): No heartbeat from core client for 30 sec - exiting 03:24:50 (5596): No heartbeat from core client for 30 sec - exiting 03:24:51 (5596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:24:52 (5596): No heartbeat from core client for 30 sec - exiting 03:24:53 (5596): No heartbeat from core client for 30 sec - exiting 03:24:54 (5596): No heartbeat from core client for 30 sec - exiting 03:24:55 (5596): No heartbeat from core client for 30 sec - exiting 03:24:56 (5596): No heartbeat from core client for 30 sec - exiting 03:24:57 (5596): No heartbeat from core client for 30 sec - exiting 03:24:58 (5596): No heartbeat from core client for 30 sec - exiting 03:24:59 (5596): No heartbeat from core client for 30 sec - exiting 03:25:00 (5596): No heartbeat from core client for 30 sec - exiting 03:25:01 (5596): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4224, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6072, iMonCtr=1 Model crash detected, will try to restart... 09:07:31 (4884): No heartbeat from core client for 30 sec - exiting 09:07:32 (4884): No heartbeat from core client for 30 sec - exiting 09:07:33 (4884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5172, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x777E997D read attempt to address 0x000080C6 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y9q0_1940_40_007615853/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Feb 2012 01:54:01	941998	13783474	hadcm3n_y9q0_1940_40_007615853_2	259,200	1,370,046	5.2857
15 Feb 2012 09:57:53	941998	13783474	hadcm3n_y9q0_1940_40_007615853_2	233,280	1,233,344	5.2870
08 Feb 2012 14:54:54	941998	13783474	hadcm3n_y9q0_1940_40_007615853_2	207,360	1,094,461	5.2781
18 Jan 2012 20:39:40	941998	13783474	hadcm3n_y9q0_1940_40_007615853_2	181,440	954,767	5.2622
08 Jan 2012 22:25:40	941998	13783474	hadcm3n_y9q0_1940_40_007615853_2	155,520	817,724	5.2580
04 Jan 2012 13:12:14	941998	13783474	hadcm3n_y9q0_1940_40_007615853_2	129,600	683,453	5.2736
29 Dec 2011 12:30:01	941998	13783474	hadcm3n_y9q0_1940_40_007615853_2	103,680	543,859	5.2456
25 Dec 2011 14:40:06	941998	13783474	hadcm3n_y9q0_1940_40_007615853_2	77,760	406,884	5.2326
21 Dec 2011 15:19:36	941998	13783474	hadcm3n_y9q0_1940_40_007615853_2	51,840	272,572	5.2579
17 Dec 2011 23:45:43	941998	13783474	hadcm3n_y9q0_1940_40_007615853_2	25,920	138,208	5.3321