Task 15667305

Name	hadcm3n_4kxw_1980_40_008325525_1
Workunit	8476660
Created	14 Mar 2013, 14:59:39 UTC
Sent	14 Mar 2013, 14:59:55 UTC
Report deadline	13 Jun 2013, 22:27:06 UTC
Received	26 Apr 2013, 6:00:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1217831
Run time	24 days 7 hours 22 min 51 sec
CPU time	15 days 9 hours 52 min 10 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:21:44 (390656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:14:58 (5140): No heartbeat from core client for 30 sec - exiting 13:14:59 (5140): No heartbeat from core client for 30 sec - exiting 13:15:00 (5140): No heartbeat from core client for 30 sec - exiting 13:15:01 (5140): No heartbeat from core client for 30 sec - exiting 13:15:02 (5140): No heartbeat from core client for 30 sec - exiting 13:15:03 (5140): No heartbeat from core client for 30 sec - exiting 13:15:04 (5140): No heartbeat from core client for 30 sec - exiting 13:15:05 (5140): No heartbeat from core client for 30 sec - exiting 13:15:06 (5140): No heartbeat from core client for 30 sec - exiting 13:15:07 (5140): No heartbeat from core client for 30 sec - exiting 13:15:08 (5140): No heartbeat from core client for 30 sec - exiting 13:15:09 (5140): No heartbeat from core client for 30 sec - exiting 13:15:10 (5140): No heartbeat from core client for 30 sec - exiting 13:15:11 (5140): No heartbeat from core client for 30 sec - exiting 13:15:12 (5140): No heartbeat from core client for 30 sec - exiting 13:15:13 (5140): No heartbeat from core client for 30 sec - exiting 13:15:14 (5140): No heartbeat from core client for 30 sec - exiting 13:15:15 (5140): No heartbeat from core client for 30 sec - exiting 13:15:16 (5140): No heartbeat from core client for 30 sec - exiting 13:15:17 (5140): No heartbeat from core client for 30 sec - exiting 13:15:18 (5140): No heartbeat from core client for 30 sec - exiting 13:15:19 (5140): No heartbeat from core client for 30 sec - exiting 13:15:20 (5140): No heartbeat from core client for 30 sec - exiting 13:15:21 (5140): No heartbeat from core client for 30 sec - exiting 13:15:22 (5140): No heartbeat from core client for 30 sec - exiting 13:15:23 (5140): No heartbeat from core client for 30 sec - exiting 13:15:24 (5140): No heartbeat from core client for 30 sec - exiting 13:15:25 (5140): No heartbeat from core client for 30 sec - exiting 13:15:26 (5140): No heartbeat from core client for 30 sec - exiting 13:15:27 (5140): No heartbeat from core client for 30 sec - exiting 13:15:28 (5140): No heartbeat from core client for 30 sec - exiting 13:15:29 (5140): No heartbeat from core client for 30 sec - exiting 13:15:30 (5140): No heartbeat from core client for 30 sec - exiting 13:15:31 (5140): No heartbeat from core client for 30 sec - exiting 13:15:32 (5140): No heartbeat from core client for 30 sec - exiting 13:15:33 (5140): No heartbeat from core client for 30 sec - exiting 13:15:34 (5140): No heartbeat from core client for 30 sec - exiting 13:15:35 (5140): No heartbeat from core client for 30 sec - exiting 13:15:36 (5140): No heartbeat from core client for 30 sec - exiting 13:15:37 (5140): No heartbeat from core client for 30 sec - exiting 13:15:38 (5140): No heartbeat from core client for 30 sec - exiting 13:15:39 (5140): No heartbeat from core client for 30 sec - exiting 13:15:40 (5140): No heartbeat from core client for 30 sec - exiting 13:15:41 (5140): No heartbeat from core client for 30 sec - exiting 13:15:42 (5140): No heartbeat from core client for 30 sec - exiting 13:15:43 (5140): No heartbeat from core client for 30 sec - exiting 13:15:44 (5140): No heartbeat from core client for 30 sec - exiting 13:15:45 (5140): No heartbeat from core client for 30 sec - exiting 13:15:46 (5140): No heartbeat from core client for 30 sec - exiting 13:15:47 (5140): No heartbeat from core client for 30 sec - exiting 13:15:48 (5140): No heartbeat from core client for 30 sec - exiting 13:15:49 (5140): No heartbeat from core client for 30 sec - exiting 13:15:50 (5140): No heartbeat from core client for 30 sec - exiting 13:15:51 (5140): No heartbeat from core client for 30 sec - exiting 13:15:52 (5140): No heartbeat from core client for 30 sec - exiting 13:15:53 (5140): No heartbeat from core client for 30 sec - exiting 13:15:54 (5140): No heartbeat from core client for 30 sec - exiting 13:15:55 (5140): No heartbeat from core client for 30 sec - exiting 13:15:56 (5140): No heartbeat from core client for 30 sec - exiting 13:15:57 (5140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:35:46 (77280): No heartbeat from core client for 30 sec - exiting 12:35:47 (77280): No heartbeat from core client for 30 sec - exiting 12:35:48 (77280): No heartbeat from core client for 30 sec - exiting 12:35:49 (77280): No heartbeat from core client for 30 sec - exiting 12:35:50 (77280): No heartbeat from core client for 30 sec - exiting 12:35:51 (77280): No heartbeat from core client for 30 sec - exiting 12:35:52 (77280): No heartbeat from core client for 30 sec - exiting 12:35:53 (77280): No heartbeat from core client for 30 sec - exiting 12:35:54 (77280): No heartbeat from core client for 30 sec - exiting 12:35:55 (77280): No heartbeat from core client for 30 sec - exiting 12:35:56 (77280): No heartbeat from core client for 30 sec - exiting 12:35:57 (77280): No heartbeat from core client for 30 sec - exiting 12:35:58 (77280): No heartbeat from core client for 30 sec - exiting 12:35:59 (77280): No heartbeat from core client for 30 sec - exiting 12:36:00 (77280): No heartbeat from core client for 30 sec - exiting 12:36:01 (77280): No heartbeat from core client for 30 sec - exiting 12:36:02 (77280): No heartbeat from core client for 30 sec - exiting 12:36:03 (77280): No heartbeat from core client for 30 sec - exiting 12:36:04 (77280): No heartbeat from core client for 30 sec - exiting 12:36:05 (77280): No heartbeat from core client for 30 sec - exiting 12:36:06 (77280): No heartbeat from core client for 30 sec - exiting 12:36:07 (77280): No heartbeat from core client for 30 sec - exiting 12:36:08 (77280): No heartbeat from core client for 30 sec - exiting 12:36:09 (77280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:09:48 (11200): No heartbeat from core client for 30 sec - exiting 14:09:49 (11200): No heartbeat from core client for 30 sec - exiting 14:09:50 (11200): No heartbeat from core client for 30 sec - exiting 14:09:51 (11200): No heartbeat from core client for 30 sec - exiting 14:09:52 (11200): No heartbeat from core client for 30 sec - exiting 14:09:53 (11200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:10:29 (54416): No heartbeat from core client for 30 sec - exiting 13:10:30 (54416): No heartbeat from core client for 30 sec - exiting 13:10:31 (54416): No heartbeat from core client for 30 sec - exiting 13:10:32 (54416): No heartbeat from core client for 30 sec - exiting 13:10:33 (54416): No heartbeat from core client for 30 sec - exiting 13:10:34 (54416): No heartbeat from core client for 30 sec - exiting 13:10:35 (54416): No heartbeat from core client for 30 sec - exiting 13:10:36 (54416): No heartbeat from core client for 30 sec - exiting 13:10:37 (54416): No heartbeat from core client for 30 sec - exiting 13:10:38 (54416): No heartbeat from core client for 30 sec - exiting 13:10:39 (54416): No heartbeat from core client for 30 sec - exiting 13:10:40 (54416): No heartbeat from core client for 30 sec - exiting 13:10:41 (54416): No heartbeat from core client for 30 sec - exiting 13:10:42 (54416): No heartbeat from core client for 30 sec - exiting 13:10:43 (54416): No heartbeat from core client for 30 sec - exiting 13:10:44 (54416): No heartbeat from core client for 30 sec - exiting 13:10:45 (54416): No heartbeat from core client for 30 sec - exiting 13:10:46 (54416): No heartbeat from core client for 30 sec - exiting 13:10:47 (54416): No heartbeat from core client for 30 sec - exiting 13:10:48 (54416): No heartbeat from core client for 30 sec - exiting 13:10:49 (54416): No heartbeat from core client for 30 sec - exiting 13:10:50 (54416): No heartbeat from core client for 30 sec - exiting 13:10:51 (54416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:28:06 (12992): No heartbeat from core client for 30 sec - exiting 19:28:07 (12992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:28:08 (12992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:58:38 (153960): No heartbeat from core client for 30 sec - exiting 12:58:39 (153960): No heartbeat from core client for 30 sec - exiting 12:58:40 (153960): No heartbeat from core client for 30 sec - exiting 12:58:41 (153960): No heartbeat from core client for 30 sec - exiting 12:58:42 (153960): No heartbeat from core client for 30 sec - exiting 12:58:43 (153960): No heartbeat from core client for 30 sec - exiting 12:58:44 (153960): No heartbeat from core client for 30 sec - exiting 12:58:45 (153960): No heartbeat from core client for 30 sec - exiting 12:58:46 (153960): No heartbeat from core client for 30 sec - exiting 12:58:47 (153960): No heartbeat from core client for 30 sec - exiting 12:58:48 (153960): No heartbeat from core client for 30 sec - exiting 12:58:49 (153960): No heartbeat from core client for 30 sec - exiting 12:58:50 (153960): No heartbeat from core client for 30 sec - exiting 12:58:51 (153960): No heartbeat from core client for 30 sec - exiting 12:58:52 (153960): No heartbeat from core client for 30 sec - exiting 12:58:53 (153960): No heartbeat from core client for 30 sec - exiting 12:58:54 (153960): No heartbeat from core client for 30 sec - exiting 12:58:55 (153960): No heartbeat from core client for 30 sec - exiting 12:58:56 (153960): No heartbeat from core client for 30 sec - exiting 12:58:57 (153960): No heartbeat from core client for 30 sec - exiting 12:58:58 (153960): No heartbeat from core client for 30 sec - exiting 12:58:59 (153960): No heartbeat from core client for 30 sec - exiting 12:59:00 (153960): No heartbeat from core client for 30 sec - exiting 12:59:01 (153960): No heartbeat from core client for 30 sec - exiting 12:59:02 (153960): No heartbeat from core client for 30 sec - exiting 12:59:03 (153960): No heartbeat from core client for 30 sec - exiting 12:59:04 (153960): No heartbeat from core client for 30 sec - exiting 12:59:05 (153960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:27:52 (5672): No heartbeat from core client for 30 sec - exiting 20:27:53 (5672): No heartbeat from core client for 30 sec - exiting 20:27:54 (5672): No heartbeat from core client for 30 sec - exiting 20:27:55 (5672): No heartbeat from core client for 30 sec - exiting 20:27:56 (5672): No heartbeat from core client for 30 sec - exiting 20:27:57 (5672): No heartbeat from core client for 30 sec - exiting 20:27:58 (5672): No heartbeat from core client for 30 sec - exiting 20:27:59 (5672): No heartbeat from core client for 30 sec - exiting 20:28:00 (5672): No heartbeat from core client for 30 sec - exiting 20:28:01 (5672): No heartbeat from core client for 30 sec - exiting 20:28:02 (5672): No heartbeat from core client for 30 sec - exiting 20:28:03 (5672): No heartbeat from core client for 30 sec - exiting 20:28:04 (5672): No heartbeat from core client for 30 sec - exiting 20:28:05 (5672): No heartbeat from core client for 30 sec - exiting 20:28:06 (5672): No heartbeat from core client for 30 sec - exiting 20:28:07 (5672): No heartbeat from core client for 30 sec - exiting 20:28:08 (5672): No heartbeat from core client for 30 sec - exiting 20:28:09 (5672): No heartbeat from core client for 30 sec - exiting 20:28:10 (5672): No heartbeat from core client for 30 sec - exiting 20:28:11 (5672): No heartbeat from core client for 30 sec - exiting 20:28:12 (5672): No heartbeat from core client for 30 sec - exiting 20:28:13 (5672): No heartbeat from core client for 30 sec - exiting 20:28:14 (5672): No heartbeat from core client for 30 sec - exiting 20:28:15 (5672): No heartbeat from core client for 30 sec - exiting 20:28:16 (5672): No heartbeat from core client for 30 sec - exiting 20:28:17 (5672): No heartbeat from core client for 30 sec - exiting 20:28:18 (5672): No heartbeat from core client for 30 sec - exiting 20:28:19 (5672): No heartbeat from core client for 30 sec - exiting 20:28:20 (5672): No heartbeat from core client for 30 sec - exiting 20:28:21 (5672): No heartbeat from core client for 30 sec - exiting 20:28:22 (5672): No heartbeat from core client for 30 sec - exiting 20:28:23 (5672): No heartbeat from core client for 30 sec - exiting 20:28:24 (5672): No heartbeat from core client for 30 sec - exiting 20:28:25 (5672): No heartbeat from core client for 30 sec - exiting 20:28:26 (5672): No heartbeat from core client for 30 sec - exiting 20:28:27 (5672): No heartbeat from core client for 30 sec - exiting 20:28:28 (5672): No heartbeat from core client for 30 sec - exiting 20:28:29 (5672): No heartbeat from core client for 30 sec - exiting 20:28:30 (5672): No heartbeat from core client for 30 sec - exiting 20:28:31 (5672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4kxwko.pjk3c10 Error converting file to netcdf: dataout/4kxwko.pik3c10 Error converting file to netcdf: dataout/4kxwko.pfk3c10 Error converting file to netcdf: dataout/4kxwka.phk3c10 Error converting file to netcdf: dataout/4kxwka.pgk3c10 Error converting file to netcdf: dataout/4kxwka.pek3c10 Error converting file to netcdf: dataout/4kxwka.pdk3c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=134232, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27656, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=36552, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 00:44:08 (28096): No heartbeat from core client for 30 sec - exiting 00:44:09 (28096): No heartbeat from core client for 30 sec - exiting 00:44:10 (28096): No heartbeat from core client for 30 sec - exiting 00:44:11 (28096): No heartbeat from core client for 30 sec - exiting 00:44:12 (28096): No heartbeat from core client for 30 sec - exiting 00:44:13 (28096): No heartbeat from core client for 30 sec - exiting 00:44:14 (28096): No heartbeat from core client for 30 sec - exiting 00:44:15 (28096): No heartbeat from core client for 30 sec - exiting 00:44:16 (28096): No heartbeat from core client for 30 sec - exiting 00:44:17 (28096): No heartbeat from core client for 30 sec - exiting 00:44:18 (28096): No heartbeat from core client for 30 sec - exiting 00:44:19 (28096): No heartbeat from core client for 30 sec - exiting 00:44:20 (28096): No heartbeat from core client for 30 sec - exiting 00:44:21 (28096): No heartbeat from core client for 30 sec - exiting 00:44:22 (28096): No heartbeat from core client for 30 sec - exiting 00:44:23 (28096): No heartbeat from core client for 30 sec - exiting 00:44:24 (28096): No heartbeat from core client for 30 sec - exiting 00:44:25 (28096): No heartbeat from core client for 30 sec - exiting 00:44:26 (28096): No heartbeat from core client for 30 sec - exiting 00:44:27 (28096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=37960, iMonCtr=1 Model crash detected, will try to restart... 10:16:19 (15128): No heartbeat from core client for 30 sec - exiting 10:16:20 (15128): No heartbeat from core client for 30 sec - exiting 10:16:21 (15128): No heartbeat from core client for 30 sec - exiting 10:16:22 (15128): No heartbeat from core client for 30 sec - exiting 10:16:23 (15128): No heartbeat from core client for 30 sec - exiting 10:16:24 (15128): No heartbeat from core client for 30 sec - exiting 10:16:25 (15128): No heartbeat from core client for 30 sec - exiting 10:16:26 (15128): No heartbeat from core client for 30 sec - exiting 10:16:27 (15128): No heartbeat from core client for 30 sec - exiting 10:16:28 (15128): No heartbeat from core client for 30 sec - exiting 10:16:29 (15128): No heartbeat from core client for 30 sec - exiting 10:16:30 (15128): No heartbeat from core client for 30 sec - exiting 10:16:31 (15128): No heartbeat from core client for 30 sec - exiting 10:16:32 (15128): No heartbeat from core client for 30 sec - exiting 10:16:33 (15128): No heartbeat from core client for 30 sec - exiting 10:16:34 (15128): No heartbeat from core client for 30 sec - exiting 10:16:35 (15128): No heartbeat from core client for 30 sec - exiting 10:16:36 (15128): No heartbeat from core client for 30 sec - exiting 10:16:37 (15128): No heartbeat from core client for 30 sec - exiting 10:16:38 (15128): No heartbeat from core client for 30 sec - exiting 10:16:39 (15128): No heartbeat from core client for 30 sec - exiting 10:16:40 (15128): No heartbeat from core client for 30 sec - exiting 10:16:41 (15128): No heartbeat from core client for 30 sec - exiting 10:16:42 (15128): No heartbeat from core client for 30 sec - exiting 10:16:43 (15128): No heartbeat from core client for 30 sec - exiting 10:16:44 (15128): No heartbeat from core client for 30 sec - exiting 10:16:45 (15128): No heartbeat from core client for 30 sec - exiting 10:16:46 (15128): No heartbeat from core client for 30 sec - exiting 10:16:47 (15128): No heartbeat from core client for 30 sec - exiting 10:16:48 (15128): No heartbeat from core client for 30 sec - exiting 10:16:49 (15128): No heartbeat from core client for 30 sec - exiting 10:16:50 (15128): No heartbeat from core client for 30 sec - exiting 10:16:51 (15128): No heartbeat from core client for 30 sec - exiting 10:16:52 (15128): No heartbeat from core client for 30 sec - exiting 10:16:53 (15128): No heartbeat from core client for 30 sec - exiting 10:16:54 (15128): No heartbeat from core client for 30 sec - exiting 10:16:55 (15128): No heartbeat from core client for 30 sec - exiting 10:16:56 (15128): No heartbeat from core client for 30 sec - exiting 10:16:57 (15128): No heartbeat from core client for 30 sec - exiting 10:16:58 (15128): No heartbeat from core client for 30 sec - exiting 10:16:59 (15128): No heartbeat from core client for 30 sec - exiting 10:17:00 (15128): No heartbeat from core client for 30 sec - exiting 10:17:01 (15128): No heartbeat from core client for 30 sec - exiting 10:17:02 (15128): No heartbeat from core client for 30 sec - exiting 10:17:03 (15128): No heartbeat from core client for 30 sec - exiting 10:17:04 (15128): No heartbeat from core client for 30 sec - exiting 10:17:05 (15128): No heartbeat from core client for 30 sec - exiting 10:17:06 (15128): No heartbeat from core client for 30 sec - exiting 10:17:07 (15128): No heartbeat from core client for 30 sec - exiting 10:17:08 (15128): No heartbeat from core client for 30 sec - exiting 10:17:09 (15128): No heartbeat from core client for 30 sec - exiting 10:17:10 (15128): No heartbeat from core client for 30 sec - exiting 10:17:11 (15128): No heartbeat from core client for 30 sec - exiting 10:17:12 (15128): No heartbeat from core client for 30 sec - exiting 10:17:13 (15128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:29:24 (85520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:46:29 (112800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:55:12 (6548): No heartbeat from core client for 30 sec - exiting 13:55:13 (6548): No heartbeat from core client for 30 sec - exiting 13:55:14 (6548): No heartbeat from core client for 30 sec - exiting 13:55:15 (6548): No heartbeat from core client for 30 sec - exiting 13:55:16 (6548): No heartbeat from core client for 30 sec - exiting 13:55:17 (6548): No heartbeat from core client for 30 sec - exiting 13:55:18 (6548): No heartbeat from core client for 30 sec - exiting 13:55:19 (6548): No heartbeat from core client for 30 sec - exiting 13:55:20 (6548): No heartbeat from core client for 30 sec - exiting 13:55:21 (6548): No heartbeat from core client for 30 sec - exiting 13:55:22 (6548): No heartbeat from core client for 30 sec - exiting 13:55:23 (6548): No heartbeat from core client for 30 sec - exiting 13:55:24 (6548): No heartbeat from core client for 30 sec - exiting 13:55:25 (6548): No heartbeat from core client for 30 sec - exiting 13:55:26 (6548): No heartbeat from core client for 30 sec - exiting 13:55:27 (6548): No heartbeat from core client for 30 sec - exiting 13:55:28 (6548): No heartbeat from core client for 30 sec - exiting 13:55:29 (6548): No heartbeat from core client for 30 sec - exiting 13:55:30 (6548): No heartbeat from core client for 30 sec - exiting 13:55:31 (6548): No heartbeat from core client for 30 sec - exiting 13:55:32 (6548): No heartbeat from core client for 30 sec - exiting 13:55:33 (6548): No heartbeat from core client for 30 sec - exiting 13:55:34 (6548): No heartbeat from core client for 30 sec - exiting 13:55:35 (6548): No heartbeat from core client for 30 sec - exiting 13:55:36 (6548): No heartbeat from core client for 30 sec - exiting 13:55:37 (6548): No heartbeat from core client for 30 sec - exiting 13:55:38 (6548): No heartbeat from core client for 30 sec - exiting 13:55:39 (6548): No heartbeat from core client for 30 sec - exiting 13:55:40 (6548): No heartbeat from core client for 30 sec - exiting 13:55:41 (6548): No heartbeat from core client for 30 sec - exiting 13:55:42 (6548): No heartbeat from core client for 30 sec - exiting 13:55:43 (6548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:42:50 (4908): No heartbeat from core client for 30 sec - exiting 15:42:51 (4908): No heartbeat from core client for 30 sec - exiting 15:42:52 (4908): No heartbeat from core client for 30 sec - exiting 15:42:53 (4908): No heartbeat from core client for 30 sec - exiting 15:42:54 (4908): No heartbeat from core client for 30 sec - exiting 15:42:55 (4908): No heartbeat from core client for 30 sec - exiting 15:42:56 (4908): No heartbeat from core client for 30 sec - exiting 15:42:57 (4908): No heartbeat from core client for 30 sec - exiting 15:42:58 (4908): No heartbeat from core client for 30 sec - exiting 15:42:59 (4908): No heartbeat from core client for 30 sec - exiting 15:43:00 (4908): No heartbeat from core client for 30 sec - exiting 15:43:01 (4908): No heartbeat from core client for 30 sec - exiting 15:43:02 (4908): No heartbeat from core client for 30 sec - exiting 15:43:03 (4908): No heartbeat from core client for 30 sec - exiting 15:43:04 (4908): No heartbeat from core client for 30 sec - exiting 15:43:05 (4908): No heartbeat from core client for 30 sec - exiting 15:43:06 (4908): No heartbeat from core client for 30 sec - exiting 15:43:07 (4908): No heartbeat from core client for 30 sec - exiting 15:43:08 (4908): No heartbeat from core client for 30 sec - exiting 15:43:09 (4908): No heartbeat from core client for 30 sec - exiting 15:43:10 (4908): No heartbeat from core client for 30 sec - exiting 15:43:11 (4908): No heartbeat from core client for 30 sec - exiting 15:43:12 (4908): No heartbeat from core client for 30 sec - exiting 15:43:13 (4908): No heartbeat from core client for 30 sec - exiting 15:43:14 (4908): No heartbeat from core client for 30 sec - exiting 15:43:15 (4908): No heartbeat from core client for 30 sec - exiting 15:43:16 (4908): No heartbeat from core client for 30 sec - exiting 15:43:17 (4908): No heartbeat from core client for 30 sec - exiting 15:43:18 (4908): No heartbeat from core client for 30 sec - exiting 15:43:19 (4908): No heartbeat from core client for 30 sec - exiting 15:43:20 (4908): No heartbeat from core client for 30 sec - exiting 15:43:21 (4908): No heartbeat from core client for 30 sec - exiting 15:43:22 (4908): No heartbeat from core client for 30 sec - exiting 15:43:23 (4908): No heartbeat from core client for 30 sec - exiting 15:43:24 (4908): No heartbeat from core client for 30 sec - exiting 15:43:25 (4908): No heartbeat from core client for 30 sec - exiting 15:43:26 (4908): No heartbeat from core client for 30 sec - exiting 15:43:27 (4908): No heartbeat from core client for 30 sec - exiting 15:43:28 (4908): No heartbeat from core client for 30 sec - exiting 15:43:29 (4908): No heartbeat from core client for 30 sec - exiting 15:43:30 (4908): No heartbeat from core client for 30 sec - exiting 15:43:31 (4908): No heartbeat from core client for 30 sec - exiting 15:43:32 (4908): No heartbeat from core client for 30 sec - exiting 15:43:33 (4908): No heartbeat from core client for 30 sec - exiting 15:43:34 (4908): No heartbeat from core client for 30 sec - exiting 15:43:35 (4908): No heartbeat from core client for 30 sec - exiting 15:43:36 (4908): No heartbeat from core client for 30 sec - exiting 15:43:37 (4908): No heartbeat from core client for 30 sec - exiting 15:43:38 (4908): No heartbeat from core client for 30 sec - exiting 15:43:39 (4908): No heartbeat from core client for 30 sec - exiting 15:43:40 (4908): No heartbeat from core client for 30 sec - exiting 15:43:41 (4908): No heartbeat from core client for 30 sec - exiting 15:43:42 (4908): No heartbeat from core client for 30 sec - exiting 15:43:43 (4908): No heartbeat from core client for 30 sec - exiting 15:43:44 (4908): No heartbeat from core client for 30 sec - exiting 15:43:45 (4908): No heartbeat from core client for 30 sec - exiting 15:43:46 (4908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:30:33 (6568): No heartbeat from core client for 30 sec - exiting 04:30:35 (6568): No heartbeat from core client for 30 sec - exiting 04:30:36 (6568): No heartbeat from core client for 30 sec - exiting 04:30:37 (6568): No heartbeat from core client for 30 sec - exiting 04:30:38 (6568): No heartbeat from core client for 30 sec - exiting 04:30:39 (6568): No heartbeat from core client for 30 sec - exiting 04:30:40 (6568): No heartbeat from core client for 30 sec - exiting 04:30:41 (6568): No heartbeat from core client for 30 sec - exiting 04:30:42 (6568): No heartbeat from core client for 30 sec - exiting 04:30:43 (6568): No heartbeat from core client for 30 sec - exiting 04:30:44 (6568): No heartbeat from core client for 30 sec - exiting 04:30:45 (6568): No heartbeat from core client for 30 sec - exiting 04:30:46 (6568): No heartbeat from core client for 30 sec - exiting 04:30:47 (6568): No heartbeat from core client for 30 sec - exiting 04:30:48 (6568): No heartbeat from core client for 30 sec - exiting 04:30:49 (6568): No heartbeat from core client for 30 sec - exiting 04:30:50 (6568): No heartbeat from core client for 30 sec - exiting 04:30:51 (6568): No heartbeat from core client for 30 sec - exiting 04:30:52 (6568): No heartbeat from core client for 30 sec - exiting 04:30:53 (6568): No heartbeat from core client for 30 sec - exiting 04:30:54 (6568): No heartbeat from core client for 30 sec - exiting 04:30:55 (6568): No heartbeat from core client for 30 sec - exiting 04:30:56 (6568): No heartbeat from core client for 30 sec - exiting 04:30:57 (6568): No heartbeat from core client for 30 sec - exiting 04:30:58 (6568): No heartbeat from core client for 30 sec - exiting 04:30:59 (6568): No heartbeat from core client for 30 sec - exiting 04:31:00 (6568): No heartbeat from core client for 30 sec - exiting 04:31:01 (6568): No heartbeat from core client for 30 sec - exiting 04:31:02 (6568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:31:03 (6568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:18:14 (75704): No heartbeat from core client for 30 sec - exiting 10:18:15 (75704): No heartbeat from core client for 30 sec - exiting 10:18:16 (75704): No heartbeat from core client for 30 sec - exiting 10:18:17 (75704): No heartbeat from core client for 30 sec - exiting 10:18:18 (75704): No heartbeat from core client for 30 sec - exiting 10:18:19 (75704): No heartbeat from core client for 30 sec - exiting 10:18:20 (75704): No heartbeat from core client for 30 sec - exiting 10:18:21 (75704): No heartbeat from core client for 30 sec - exiting 10:18:22 (75704): No heartbeat from core client for 30 sec - exiting 10:18:23 (75704): No heartbeat from core client for 30 sec - exiting 10:18:24 (75704): No heartbeat from core client for 30 sec - exiting 10:18:25 (75704): No heartbeat from core client for 30 sec - exiting 10:18:26 (75704): No heartbeat from core client for 30 sec - exiting 10:18:27 (75704): No heartbeat from core client for 30 sec - exiting 10:18:28 (75704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:16:10 (2988): No heartbeat from core client for 30 sec - exiting 15:16:11 (2988): No heartbeat from core client for 30 sec - exiting 15:16:12 (2988): No heartbeat from core client for 30 sec - exiting 15:16:13 (2988): No heartbeat from core client for 30 sec - exiting 15:16:14 (2988): No heartbeat from core client for 30 sec - exiting 15:16:15 (2988): No heartbeat from core client for 30 sec - exiting 15:16:16 (2988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=60300, iMonCtr=1 Model crash detected, will try to restart... 12:54:33 (24752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:03:26 (78332): No heartbeat from core client for 30 sec - exiting 06:03:27 (78332): No heartbeat from core client for 30 sec - exiting 06:03:28 (78332): No heartbeat from core client for 30 sec - exiting 06:03:29 (78332): No heartbeat from core client for 30 sec - exiting 06:03:30 (78332): No heartbeat from core client for 30 sec - exiting 06:03:31 (78332): No heartbeat from core client for 30 sec - exiting 06:03:32 (78332): No heartbeat from core client for 30 sec - exiting 06:03:33 (78332): No heartbeat from core client for 30 sec - exiting 06:03:34 (78332): No heartbeat from core client for 30 sec - exiting 06:03:35 (78332): No heartbeat from core client for 30 sec - exiting 06:03:36 (78332): No heartbeat from core client for 30 sec - exiting 06:03:37 (78332): No heartbeat from core client for 30 sec - exiting 06:03:38 (78332): No heartbeat from core client for 30 sec - exiting 06:03:39 (78332): No heartbeat from core client for 30 sec - exiting 06:03:40 (78332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Apr 2013 04:31:40	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	1,036,800	1,331,529	1.2843
25 Apr 2013 14:20:29	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	1,010,880	1,298,233	1.2843
24 Apr 2013 17:23:49	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	984,960	1,264,443	1.2838
23 Apr 2013 02:06:54	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	959,040	1,231,014	1.2836
21 Apr 2013 17:51:00	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	933,120	1,197,709	1.2836
20 Apr 2013 23:54:25	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	907,200	1,164,791	1.2839
20 Apr 2013 03:22:12	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	881,280	1,130,993	1.2834
19 Apr 2013 03:11:46	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	855,360	1,097,108	1.2826
18 Apr 2013 07:10:43	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	829,440	1,062,602	1.2811
17 Apr 2013 12:38:16	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	803,520	1,028,789	1.2804
16 Apr 2013 13:11:10	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	777,600	995,067	1.2797
15 Apr 2013 19:12:26	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	751,680	961,126	1.2786
14 Apr 2013 23:02:15	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	725,760	927,608	1.2781
14 Apr 2013 04:25:07	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	699,840	893,776	1.2771
13 Apr 2013 12:14:37	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	673,920	860,328	1.2766
12 Apr 2013 20:12:43	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	648,000	827,172	1.2765
11 Apr 2013 22:15:25	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	622,080	793,579	1.2757
11 Apr 2013 07:13:31	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	596,160	760,245	1.2752
10 Apr 2013 12:54:39	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	570,240	727,412	1.2756
09 Apr 2013 13:45:09	1217831	15667305	hadcm3n_4kxw_1980_40_008325525_1	544,320	693,955	1.2749