Task 16017811

Name	hadcm3n_7xje_1980_40_008454749_2
Workunit	8605605
Created	15 Sep 2013, 13:50:42 UTC
Sent	15 Sep 2013, 14:23:02 UTC
Report deadline	15 Dec 2013, 21:50:13 UTC
Received	20 Nov 2013, 18:36:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1227579
Run time	6 days 2 hours 38 min 40 sec
CPU time	5 days 21 hours 1 min 22 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.89 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6740, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5340, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5820, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=180, iMonCtr=1 Model crash detected, will try to restart... 18:30:26 (4932): No heartbeat from core client for 30 sec - exiting 18:30:27 (4932): No heartbeat from core client for 30 sec - exiting 18:30:28 (4932): No heartbeat from core client for 30 sec - exiting 18:30:29 (4932): No heartbeat from core client for 30 sec - exiting 18:30:30 (4932): No heartbeat from core client for 30 sec - exiting 18:30:31 (4932): No heartbeat from core client for 30 sec - exiting 18:30:32 (4932): No heartbeat from core client for 30 sec - exiting 18:30:33 (4932): No heartbeat from core client for 30 sec - exiting 18:30:34 (4932): No heartbeat from core client for 30 sec - exiting 18:30:35 (4932): No heartbeat from core client for 30 sec - exiting 18:30:36 (4932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... C13:58:54 (4512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:34:42 (384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5504, iMonCtr=1 Model crash detected, will try to restart... 13:41:06 (4628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6004, iMonCtr=1 Model crash detected, will try to restart... 14:51:20 (4756): No heartbeat from core client for 30 sec - exiting 14:51:21 (4756): No heartbeat from core client for 30 sec - exiting 14:51:22 (4756): No heartbeat from core client for 30 sec - exiting 14:51:23 (4756): No heartbeat from core client for 30 sec - exiting 14:51:24 (4756): No heartbeat from core client for 30 sec - exiting 14:51:25 (4756): No heartbeat from core client for 30 sec - exiting 14:51:26 (4756): No heartbeat from core client for 30 sec - exiting 14:51:27 (4756): No heartbeat from core client for 30 sec - exiting 14:51:28 (4756): No heartbeat from core client for 30 sec - exiting 14:51:29 (4756): No heartbeat from core client for 30 sec - exiting 14:51:30 (4756): No heartbeat from core client for 30 sec - exiting 14:51:31 (4756): No heartbeat from core client for 30 sec - exiting 14:51:33 (4756): No heartbeat from core client for 30 sec - exiting 14:51:34 (4756): No heartbeat from core client for 30 sec - exiting 14:51:35 (4756): No heartbeat from core client for 30 sec - exiting 14:51:36 (4756): No heartbeat from core client for 30 sec - exiting 14:51:37 (4756): No heartbeat from core client for 30 sec - exiting 14:51:38 (4756): No heartbeat from core client for 30 sec - exiting 14:51:39 (4756): No heartbeat from core client for 30 sec - exiting 14:51:40 (4756): No heartbeat from core client for 30 sec - exiting 14:51:41 (4756): No heartbeat from core client for 30 sec - exiting 14:51:42 (4756): No heartbeat from core client for 30 sec - exiting 14:51:43 (4756): No heartbeat from core client for 30 sec - exiting 14:51:44 (4756): No heartbeat from core client for 30 sec - exiting 14:51:45 (4756): No heartbeat from core client for 30 sec - exiting 14:51:46 (4756): No heartbeat from core client for 30 sec - exiting 14:51:47 (4756): No heartbeat from core client for 30 sec - exiting 14:51:48 (4756): No heartbeat from core client for 30 sec - exiting 14:51:49 (4756): No heartbeat from core client for 30 sec - exiting 14:51:50 (4756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:03:12 (4596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:25 (5076): No heartbeat from core client for 30 sec - exiting 08:40:26 (5076): No heartbeat from core client for 30 sec - exiting 08:40:27 (5076): No heartbeat from core client for 30 sec - exiting 08:40:29 (5076): No heartbeat from core client for 30 sec - exiting 08:40:30 (5076): No heartbeat from core client for 30 sec - exiting 08:40:31 (5076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:32 (5076): No heartbeat from core client for 30 sec - exiting 18:33:46 (4188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:17:22 (4556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:15:48 (5072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:32:04 (4484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:04:43 (4644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:02:50 (4288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:51 (4456): No heartbeat from core client for 30 sec - exiting 18:31:52 (4456): No heartbeat from core client for 30 sec - exiting 18:31:53 (4456): No heartbeat from core client for 30 sec - exiting 18:31:54 (4456): No heartbeat from core client for 30 sec - exiting 18:31:55 (4456): No heartbeat from core client for 30 sec - exiting 18:31:56 (4456): No heartbeat from core client for 30 sec - exiting 18:31:57 (4456): No heartbeat from core client for 30 sec - exiting 18:31:58 (4456): No heartbeat from core client for 30 sec - exiting 18:31:59 (4456): No heartbeat from core client for 30 sec - exiting 18:32:01 (4456): No heartbeat from core client for 30 sec - exiting 18:32:02 (4456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2500, iMonCtr=1 Model crash detected, will try to restart... 18:25:55 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5072, iMonCtr=1 Model crash detected, will try to restart... 18:35:30 (4084): No heartbeat from core client for 30 sec - exiting 18:35:31 (4084): No heartbeat from core client for 30 sec - exiting 18:35:32 (4084): No heartbeat from core client for 30 sec - exiting 18:35:33 (4084): No heartbeat from core client for 30 sec - exiting 18:35:34 (4084): No heartbeat from core client for 30 sec - exiting 18:35:35 (4084): No heartbeat from core client for 30 sec - exiting 18:35:36 (4084): No heartbeat from core client for 30 sec - exiting 18:35:37 (4084): No heartbeat from core client for 30 sec - exiting 18:35:38 (4084): No heartbeat from core client for 30 sec - exiting 18:35:39 (4084): No heartbeat from core client for 30 sec - exiting 18:35:40 (4084): No heartbeat from core client for 30 sec - exiting 18:35:41 (4084): No heartbeat from core client for 30 sec - exiting 18:35:42 (4084): No heartbeat from core client for 30 sec - exiting 18:35:43 (4084): No heartbeat from core client for 30 sec - exiting 18:35:44 (4084): No heartbeat from core client for 30 sec - exiting 18:35:45 (4084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:05:18 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:31 (3940): No heartbeat from core client for 30 sec - exiting 18:25:32 (3940): No heartbeat from core client for 30 sec - exiting 18:25:33 (3940): No heartbeat from core client for 30 sec - exiting 18:25:34 (3940): No heartbeat from core client for 30 sec - exiting 18:25:35 (3940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:36 (3940): No heartbeat from core client for 30 sec - exiting 18:34:40 (5092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5604, iMonCtr=1 Model crash detected, will try to restart... 13:12:17 (3888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2568, iMonCtr=1 Model crash detected, will try to restart... 13:51:08 (4360): No heartbeat from core client for 30 sec - exiting 13:51:10 (4360): No heartbeat from core client for 30 sec - exiting 13:51:11 (4360): No heartbeat from core client for 30 sec - exiting 13:51:12 (4360): No heartbeat from core client for 30 sec - exiting 13:51:13 (4360): No heartbeat from core client for 30 sec - exiting 13:51:14 (4360): No heartbeat from core client for 30 sec - exiting 13:51:15 (4360): No heartbeat from core client for 30 sec - exiting 13:51:16 (4360): No heartbeat from core client for 30 sec - exiting 13:51:17 (4360): No heartbeat from core client for 30 sec - exiting 13:51:18 (4360): No heartbeat from core client for 30 sec - exiting 13:51:19 (4360): No heartbeat from core client for 30 sec - exiting 13:51:20 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:11:35 (4552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:33:33 (5032): No heartbeat from core client for 30 sec - exiting 18:33:34 (5032): No heartbeat from core client for 30 sec - exiting 18:33:36 (5032): No heartbeat from core client for 30 sec - exiting 18:33:37 (5032): No heartbeat from core client for 30 sec - exiting 18:33:38 (5032): No heartbeat from core client for 30 sec - exiting 18:33:39 (5032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:41:11 (4420): No heartbeat from core client for 30 sec - exiting 18:41:12 (4420): No heartbeat from core client for 30 sec - exiting 18:41:13 (4420): No heartbeat from core client for 30 sec - exiting 18:41:14 (4420): No heartbeat from core client for 30 sec - exiting 18:41:15 (4420): No heartbeat from core client for 30 sec - exiting 18:41:16 (4420): No heartbeat from core client for 30 sec - exiting 18:41:17 (4420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3432, iMonCtr=1 Model crash detected, will try to restart... 18:29:43 (4636): No heartbeat from core client for 30 sec - exiting 18:29:44 (4636): No heartbeat from core client for 30 sec - exiting 18:29:45 (4636): No heartbeat from core client for 30 sec - exiting 18:29:46 (4636): No heartbeat from core client for 30 sec - exiting 18:29:47 (4636): No heartbeat from core client for 30 sec - exiting 18:29:48 (4636): No heartbeat from core client for 30 sec - exiting 18:29:49 (4636): No heartbeat from core client for 30 sec - exiting 18:29:50 (4636): No heartbeat from core client for 30 sec - exiting 18:29:51 (4636): No heartbeat from core client for 30 sec - exiting 18:29:52 (4636): No heartbeat from core client for 30 sec - exiting 18:29:53 (4636): No heartbeat from core client for 30 sec - exiting 18:29:54 (4636): No heartbeat from core client for 30 sec - exiting 18:29:55 (4636): No heartbeat from core client for 30 sec - exiting 18:29:56 (4636): No heartbeat from core client for 30 sec - exiting 18:29:57 (4636): No heartbeat from core client for 30 sec - exiting 18:29:58 (4636): No heartbeat from core client for 30 sec - exiting 18:29:59 (4636): No heartbeat from core client for 30 sec - exiting 18:30:00 (4636): No heartbeat from core client for 30 sec - exiting 18:30:01 (4636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:02 (4636): No heartbeat from core client for 30 sec - exiting 18:22:39 (264): No heartbeat from core client for 30 sec - exiting 18:22:41 (264): No heartbeat from core client for 30 sec - exiting 18:22:42 (264): No heartbeat from core client for 30 sec - exiting 18:22:43 (264): No heartbeat from core client for 30 sec - exiting 18:22:44 (264): No heartbeat from core client for 30 sec - exiting 18:22:45 (264): No heartbeat from core client for 30 sec - exiting 18:22:46 (264): No heartbeat from core client for 30 sec - exiting 18:22:47 (264): No heartbeat from core client for 30 sec - exiting 18:22:48 (264): No heartbeat from core client for 30 sec - exiting 18:22:49 (264): No heartbeat from core client for 30 sec - exiting 18:22:50 (264): No heartbeat from core client for 30 sec - exiting 18:22:51 (264): No heartbeat from core client for 30 sec - exiting 18:22:52 (264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 14:18:49 (1856): No heartbeat from core client for 30 sec - exiting 14:18:50 (1856): No heartbeat from core client for 30 sec - exiting 14:18:51 (1856): No heartbeat from core client for 30 sec - exiting 14:18:52 (1856): No heartbeat from core client for 30 sec - exiting 14:18:53 (1856): No heartbeat from core client for 30 sec - exiting 14:18:54 (1856): No heartbeat from core client for 30 sec - exiting 14:18:55 (1856): No heartbeat from core client for 30 sec - exiting 14:18:56 (1856): No heartbeat from core client for 30 sec - exiting 14:18:57 (1856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1556, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4164, iMonCtr=1 Model crash detected, will try to restart... 13:38:23 (3720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:11:47 (5108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:54:44 (3364): No heartbeat from core client for 30 sec - exiting 18:54:45 (3364): No heartbeat from core client for 30 sec - exiting 18:54:46 (3364): No heartbeat from core client for 30 sec - exiting 18:54:47 (3364): No heartbeat from core client for 30 sec - exiting 18:54:48 (3364): No heartbeat from core client for 30 sec - exiting 18:54:49 (3364): No heartbeat from core client for 30 sec - exiting 18:54:50 (3364): No heartbeat from core client for 30 sec - exiting 18:54:51 (3364): No heartbeat from core client for 30 sec - exiting 18:54:52 (3364): No heartbeat from core client for 30 sec - exiting 18:54:53 (3364): No heartbeat from core client for 30 sec - exiting 18:54:54 (3364): No heartbeat from core client for 30 sec - exiting 18:54:55 (3364): No heartbeat from core client for 30 sec - exiting 18:54:56 (3364): No heartbeat from core client for 30 sec - exiting 18:54:57 (3364): No heartbeat from core client for 30 sec - exiting 18:54:58 (3364): No heartbeat from core client for 30 sec - exiting 18:54:59 (3364): No heartbeat from core client for 30 sec - exiting 18:55:00 (3364): No heartbeat from core client for 30 sec - exiting 18:55:01 (3364): No heartbeat from core client for 30 sec - exiting 18:55:02 (3364): No heartbeat from core client for 30 sec - exiting 18:55:03 (3364): No heartbeat from core client for 30 sec - exiting 18:55:04 (3364): No heartbeat from core client for 30 sec - exiting 18:55:05 (3364): No heartbeat from core client for 30 sec - exiting 18:55:06 (3364): No heartbeat from core client for 30 sec - exiting 18:55:07 (3364): No heartbeat from core client for 30 sec - exiting 18:55:08 (3364): No heartbeat from core client for 30 sec - exiting 18:55:09 (3364): No heartbeat from core client for 30 sec - exiting 18:55:10 (3364): No heartbeat from core client for 30 sec - exiting 18:55:11 (3364): No heartbeat from core client for 30 sec - exiting 18:55:12 (3364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:46:54 (3364): No heartbeat from core client for 30 sec - exiting 08:46:55 (3364): No heartbeat from core client for 30 sec - exiting 08:46:56 (3364): No heartbeat from core client for 30 sec - exiting 08:46:57 (3364): No heartbeat from core client for 30 sec - exiting 08:46:58 (3364): No heartbeat from core client for 30 sec - exiting 08:46:59 (3364): No heartbeat from core client for 30 sec - exiting 08:47:00 (3364): No heartbeat from core client for 30 sec - exiting 08:47:01 (3364): No heartbeat from core client for 30 sec - exiting 08:47:02 (3364): No heartbeat from core client for 30 sec - exiting 08:47:03 (3364): No heartbeat from core client for 30 sec - exiting 08:47:04 (3364): No heartbeat from core client for 30 sec - exiting 08:47:05 (3364): No heartbeat from core client for 30 sec - exiting 08:47:06 (3364): No heartbeat from core client for 30 sec - exiting 08:47:07 (3364): No heartbeat from core client for 30 sec - exiting 08:47:08 (3364): No heartbeat from core client for 30 sec - exiting 08:47:09 (3364): No heartbeat from core client for 30 sec - exiting 08:47:10 (3364): No heartbeat from core client for 30 sec - exiting 08:47:11 (3364): No heartbeat from core client for 30 sec - exiting 08:47:12 (3364): No heartbeat from core client for 30 sec - exiting 08:47:13 (3364): No heartbeat from core client for 30 sec - exiting 08:47:14 (3364): No heartbeat from core client for 30 sec - exiting 08:47:15 (3364): No heartbeat from core client for 30 sec - exiting 08:47:16 (3364): No heartbeat from core client for 30 sec - exiting 08:47:17 (3364): No heartbeat from core client for 30 sec - exiting 08:47:18 (3364): No heartbeat from core client for 30 sec - exiting 08:47:19 (3364): No heartbeat from core client for 30 sec - exiting 08:47:20 (3364): No heartbeat from core client for 30 sec - exiting 08:47:21 (3364): No heartbeat from core client for 30 sec - exiting 08:47:22 (3364): No heartbeat from core client for 30 sec - exiting 08:47:23 (3364): No heartbeat from core client for 30 sec - exiting 08:47:24 (3364): No heartbeat from core client for 30 sec - exiting 08:47:25 (3364): No heartbeat from core client for 30 sec - exiting 08:47:26 (3364): No heartbeat from core client for 30 sec - exiting 08:47:27 (3364): No heartbeat from core client for 30 sec - exiting 08:47:28 (3364): No heartbeat from core client for 30 sec - exiting 08:47:29 (3364): No heartbeat from core client for 30 sec - exiting 08:47:30 (3364): No heartbeat from core client for 30 sec - exiting 08:47:31 (3364): No heartbeat from core client for 30 sec - exiting 08:47:32 (3364): No heartbeat from core client for 30 sec - exiting 08:47:33 (3364): No heartbeat from core client for 30 sec - exiting 08:47:34 (3364): No heartbeat from core client for 30 sec - exiting 08:47:35 (3364): No heartbeat from core client for 30 sec - exiting 08:47:36 (3364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:13:56 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:09:36 (1440): No heartbeat from core client for 30 sec - exiting 14:09:37 (1440): No heartbeat from core client for 30 sec - exiting 14:09:38 (1440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:07:27 (5468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:46 (4420): No heartbeat from core client for 30 sec - exiting 18:34:47 (4420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:21:50 (4276): No heartbeat from core client for 30 sec - exiting 18:21:51 (4276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:33:39 (4500): No heartbeat from core client for 30 sec - exiting 18:33:40 (4500): No heartbeat from core client for 30 sec - exiting 18:33:41 (4500): No heartbeat from core client for 30 sec - exiting 18:33:42 (4500): No heartbeat from core client for 30 sec - exiting 18:33:43 (4500): No heartbeat from core client for 30 sec - exiting 18:33:44 (4500): No heartbeat from core client for 30 sec - exiting 18:33:45 (4500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:34:35 (5072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:41:57 (4956): No heartbeat from core client for 30 sec - exiting 18:41:58 (4956): No heartbeat from core client for 30 sec - exiting 18:41:59 (4956): No heartbeat from core client for 30 sec - exiting 18:42:00 (4956): No heartbeat from core client for 30 sec - exiting 18:42:01 (4956): No heartbeat from core client for 30 sec - exiting 18:42:02 (4956): No heartbeat from core client for 30 sec - exiting 18:42:03 (4956): No heartbeat from core client for 30 sec - exiting 18:42:04 (4956): No heartbeat from core client for 30 sec - exiting 18:42:05 (4956): No heartbeat from core client for 30 sec - exiting 18:42:06 (4956): No heartbeat from core client for 30 sec - exiting 18:42:07 (4956): No heartbeat from core client for 30 sec - exiting 18:42:08 (4956): No heartbeat from core client for 30 sec - exiting 18:42:09 (4956): No heartbeat from core client for 30 sec - exiting 18:42:10 (4956): No heartbeat from core client for 30 sec - exiting 18:42:11 (4956): No heartbeat from core client for 30 sec - exiting 18:42:12 (4956): No heartbeat from core client for 30 sec - exiting 18:42:13 (4956): No heartbeat from core client for 30 sec - exiting 18:42:14 (4956): No heartbeat from core client for 30 sec - exiting 18:42:15 (4956): No heartbeat from core client for 30 sec - exiting 18:42:16 (4956): No heartbeat from core client for 30 sec - exiting 18:42:17 (4956): No heartbeat from core client for 30 sec - exiting 18:42:18 (4956): No heartbeat from core client for 30 sec - exiting 18:42:19 (4956): No heartbeat from core client for 30 sec - exiting 18:42:20 (4956): No heartbeat from core client for 30 sec - exiting 18:42:21 (4956): No heartbeat from core client for 30 sec - exiting 18:42:22 (4956): No heartbeat from core client for 30 sec - exiting 18:42:23 (4956): No heartbeat from core client for 30 sec - exiting 18:42:24 (4956): No heartbeat from core client for 30 sec - exiting 18:42:25 (4956): No heartbeat from core client for 30 sec - exiting 18:42:26 (4956): No heartbeat from core client for 30 sec - exiting 18:42:27 (4956): No heartbeat from core client for 30 sec - exiting 18:42:28 (4956): No heartbeat from core client for 30 sec - exiting 18:42:29 (4956): No heartbeat from core client for 30 sec - exiting 18:42:30 (4956): No heartbeat from core client for 30 sec - exiting 18:42:31 (4956): No heartbeat from core client for 30 sec - exiting 18:42:32 (4956): No heartbeat from core client for 30 sec - exiting 18:42:33 (4956): No heartbeat from core client for 30 sec - exiting 18:42:34 (4956): No heartbeat from core client for 30 sec - exiting 18:42:35 (4956): No heartbeat from core client for 30 sec - exiting 18:42:36 (4956): No heartbeat from core client for 30 sec - exiting 18:42:37 (4956): No heartbeat from core client for 30 sec - exiting 18:42:38 (4956): No heartbeat from core client for 30 sec - exiting 18:42:39 (4956): No heartbeat from core client for 30 sec - exiting 18:42:40 (4956): No heartbeat from core client for 30 sec - exiting 18:42:41 (4956): No heartbeat from core client for 30 sec - exiting 18:42:42 (4956): No heartbeat from core client for 30 sec - exiting 18:42:43 (4956): No heartbeat from core client for 30 sec - exiting 18:42:44 (4956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5424, iMonCtr=1 Model crash detected, will try to restart... 13:27:23 (4504): No heartbeat from core client for 30 sec - exiting 13:27:24 (4504): No heartbeat from core client for 30 sec - exiting 13:27:25 (4504): No heartbeat from core client for 30 sec - exiting 13:27:26 (4504): No heartbeat from core client for 30 sec - exiting 13:27:27 (4504): No heartbeat from core client for 30 sec - exiting 13:27:28 (4504): No heartbeat from core client for 30 sec - exiting 13:27:29 (4504): No heartbeat from core client for 30 sec - exiting 13:27:30 (4504): No heartbeat from core client for 30 sec - exiting 13:27:31 (4504): No heartbeat from core client for 30 sec - exiting 13:27:32 (4504): No heartbeat from core client for 30 sec - exiting 13:27:33 (4504): No heartbeat from core client for 30 sec - exiting 13:27:34 (4504): No heartbeat from core client for 30 sec - exiting 13:27:35 (4504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:48 (5216): No heartbeat from core client for 30 sec - exiting 18:43:49 (5216): No heartbeat from core client for 30 sec - exiting 18:43:50 (5216): No heartbeat from core client for 30 sec - exiting 18:43:51 (5216): No heartbeat from core client for 30 sec - exiting 18:43:52 (5216): No heartbeat from core client for 30 sec - exiting 18:43:53 (5216): No heartbeat from core client for 30 sec - exiting 18:43:54 (5216): No heartbeat from core client for 30 sec - exiting 18:43:56 (5216): No heartbeat from core client for 30 sec - exiting 18:43:57 (5216): No heartbeat from core client for 30 sec - exiting 18:43:58 (5216): No heartbeat from core client for 30 sec - exiting 18:43:59 (5216): No heartbeat from core client for 30 sec - exiting 18:44:00 (5216): No heartbeat from core client for 30 sec - exiting 18:44:01 (5216): No heartbeat from core client for 30 sec - exiting 18:44:02 (5216): No heartbeat from core client for 30 sec - exiting 18:44:03 (5216): No heartbeat from core client for 30 sec - exiting 18:44:04 (5216): No heartbeat from core client for 30 sec - exiting 18:44:05 (5216): No heartbeat from core client for 30 sec - exiting 18:44:06 (5216): No heartbeat from core client for 30 sec - exiting 18:44:07 (5216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:19:55 (5248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:35:29 (3736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:09:43 (2472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:03:14 (5408): No heartbeat from core client for 30 sec - exiting 09:03:15 (5408): No heartbeat from core client for 30 sec - exiting 09:03:16 (5408): No heartbeat from core client for 30 sec - exiting 09:03:17 (5408): No heartbeat from core client for 30 sec - exiting 09:03:18 (5408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3348, iMonCtr=1 Model crash detected, will try to restart... 15:04:19 (5440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:42 (3752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:15:09 (4072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:15:15 (4012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4192, iMonCtr=1 Model crash detected, will try to restart... 13:36:19 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4344, iMonCtr=1 Model crash detected, will try to restart... 18:35:10 (3168): No heartbeat from core client for 30 sec - exiting 18:35:11 (3168): No heartbeat from core client for 30 sec - exiting 18:35:12 (3168): No heartbeat from core client for 30 sec - exiting 18:35:13 (3168): No heartbeat from core client for 30 sec - exiting 18:35:14 (3168): No heartbeat from core client for 30 sec - exiting 18:35:15 (3168): No heartbeat from core client for 30 sec - exiting 18:35:16 (3168): No heartbeat from core client for 30 sec - exiting 18:35:17 (3168): No heartbeat from core client for 30 sec - exiting 18:35:18 (3168): No heartbeat from core client for 30 sec - exiting 18:35:19 (3168): No heartbeat from core client for 30 sec - exiting 18:35:20 (3168): No heartbeat from core client for 30 sec - exiting 18:35:21 (3168): No heartbeat from core client for 30 sec - exiting 18:35:22 (3168): No heartbeat from core client for 30 sec - exiting 18:35:23 (3168): No heartbeat from core client for 30 sec - exiting 18:35:24 (3168): No heartbeat from core client for 30 sec - exiting 18:35:25 (3168): No heartbeat from core client for 30 sec - exiting 18:35:26 (3168): No heartbeat from core client for 30 sec - exiting 18:35:28 (3168): No heartbeat from core client for 30 sec - exiting 18:35:29 (3168): No heartbeat from core client for 30 sec - exiting 18:35:30 (3168): No heartbeat from core client for 30 sec - exiting 18:35:31 (3168): No heartbeat from core client for 30 sec - exiting 18:35:32 (3168): No heartbeat from core client for 30 sec - exiting 18:35:33 (3168): No heartbeat from core client for 30 sec - exiting 18:35:34 (3168): No heartbeat from core client for 30 sec - exiting 18:35:35 (3168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:36 (3168): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77017383 read attempt to address 0x40BA98ED Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x776B3AC3 read attempt to address 0x40BA98F5 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_7xje_1980_40_008454749/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Nov 2013 20:16:21	1227579	16017811	hadcm3n_7xje_1980_40_008454749_2	259,200	506,611	1.9545
14 Nov 2013 14:52:41	1227579	16017811	hadcm3n_7xje_1980_40_008454749_2	233,280	456,540	1.9570
08 Nov 2013 15:04:47	1227579	16017811	hadcm3n_7xje_1980_40_008454749_2	207,360	402,226	1.9397
29 Oct 2013 20:13:30	1227579	16017811	hadcm3n_7xje_1980_40_008454749_2	181,440	347,032	1.9127
25 Oct 2013 08:18:06	1227579	16017811	hadcm3n_7xje_1980_40_008454749_2	155,520	297,399	1.9123
24 Oct 2013 19:41:18	1227579	16017811	hadcm3n_7xje_1980_40_008454749_2	129,600	252,483	1.9482
15 Oct 2013 19:25:18	1227579	16017811	hadcm3n_7xje_1980_40_008454749_2	103,680	198,764	1.9171
07 Oct 2013 18:19:19	1227579	16017811	hadcm3n_7xje_1980_40_008454749_2	77,760	149,926	1.9281
30 Sep 2013 19:08:58	1227579	16017811	hadcm3n_7xje_1980_40_008454749_2	51,840	99,601	1.9213
25 Sep 2013 09:23:04	1227579	16017811	hadcm3n_7xje_1980_40_008454749_2	25,920	49,820	1.9221