Task 15701858

Name	hadcm3n_3bqy_1980_40_008320699_2
Workunit	8471834
Created	4 Apr 2013, 13:35:33 UTC
Sent	4 Apr 2013, 13:35:58 UTC
Report deadline	4 Jul 2013, 21:03:09 UTC
Received	11 Jun 2013, 11:29:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1277214
Run time	16 days 22 hours 2 min 9 sec
CPU time	15 days 18 hours 54 min 46 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.55 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:25:53 (6688): No heartbeat from core client for 30 sec - exiting 21:25:54 (6688): No heartbeat from core client for 30 sec - exiting 21:25:55 (6688): No heartbeat from core client for 30 sec - exiting 21:25:56 (6688): No heartbeat from core client for 30 sec - exiting 21:25:58 (6688): No heartbeat from core client for 30 sec - exiting 21:25:59 (6688): No heartbeat from core client for 30 sec - exiting 21:26:00 (6688): No heartbeat from core client for 30 sec - exiting 21:26:01 (6688): No heartbeat from core client for 30 sec - exiting 21:26:02 (6688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:11:23 (5772): No heartbeat from core client for 30 sec - exiting 08:11:24 (5772): No heartbeat from core client for 30 sec - exiting 08:11:26 (5772): No heartbeat from core client for 30 sec - exiting 08:11:27 (5772): No heartbeat from core client for 30 sec - exiting 08:11:28 (5772): No heartbeat from core client for 30 sec - exiting 08:11:29 (5772): No heartbeat from core client for 30 sec - exiting 08:11:30 (5772): No heartbeat from core client for 30 sec - exiting 08:11:31 (5772): No heartbeat from core client for 30 sec - exiting 08:11:32 (5772): No heartbeat from core client for 30 sec - exiting 08:11:33 (5772): No heartbeat from core client for 30 sec - exiting 08:11:34 (5772): No heartbeat from core client for 30 sec - exiting 08:11:35 (5772): No heartbeat from core client for 30 sec - exiting 08:11:36 (5772): No heartbeat from core client for 30 sec - exiting 08:11:37 (5772): No heartbeat from core client for 30 sec - exiting 08:11:38 (5772): No heartbeat from core client for 30 sec - exiting 08:11:39 (5772): No heartbeat from core client for 30 sec - exiting 08:11:40 (5772): No heartbeat from core client for 30 sec - exiting 08:11:42 (5772): No heartbeat from core client for 30 sec - exiting 08:11:43 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:17:43 (4968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:54:55 (4616): No heartbeat from core client for 30 sec - exiting 15:54:56 (4616): No heartbeat from core client for 30 sec - exiting 15:54:57 (4616): No heartbeat from core client for 30 sec - exiting 15:54:59 (4616): No heartbeat from core client for 30 sec - exiting 15:55:00 (4616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:31:55 (6776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:02:39 (5576): No heartbeat from core client for 30 sec - exiting 11:02:40 (5576): No heartbeat from core client for 30 sec - exiting 11:02:41 (5576): No heartbeat from core client for 30 sec - exiting 11:02:42 (5576): No heartbeat from core client for 30 sec - exiting 11:02:43 (5576): No heartbeat from core client for 30 sec - exiting 11:02:44 (5576): No heartbeat from core client for 30 sec - exiting 11:02:45 (5576): No heartbeat from core client for 30 sec - exiting 11:02:47 (5576): No heartbeat from core client for 30 sec - exiting 11:02:48 (5576): No heartbeat from core client for 30 sec - exiting 11:02:49 (5576): No heartbeat from core client for 30 sec - exiting 11:02:50 (5576): No heartbeat from core client for 30 sec - exiting 11:02:51 (5576): No heartbeat from core client for 30 sec - exiting 11:02:52 (5576): No heartbeat from core client for 30 sec - exiting 11:02:53 (5576): No heartbeat from core client for 30 sec - exiting 11:02:54 (5576): No heartbeat from core client for 30 sec - exiting 11:02:55 (5576): No heartbeat from core client for 30 sec - exiting 11:02:56 (5576): No heartbeat from core client for 30 sec - exiting 11:02:57 (5576): No heartbeat from core client for 30 sec - exiting 11:02:59 (5576): No heartbeat from core client for 30 sec - exiting 11:03:00 (5576): No heartbeat from core client for 30 sec - exiting 11:03:01 (5576): No heartbeat from core client for 30 sec - exiting 11:03:02 (5576): No heartbeat from core client for 30 sec - exiting 11:03:03 (5576): No heartbeat from core client for 30 sec - exiting 11:03:04 (5576): No heartbeat from core client for 30 sec - exiting 11:03:05 (5576): No heartbeat from core client for 30 sec - exiting 11:03:06 (5576): No heartbeat from core client for 30 sec - exiting 11:03:07 (5576): No heartbeat from core client for 30 sec - exiting 11:03:09 (5576): No heartbeat from core client for 30 sec - exiting 11:03:10 (5576): No heartbeat from core client for 30 sec - exiting 11:03:11 (5576): No heartbeat from core client for 30 sec - exiting 11:03:12 (5576): No heartbeat from core client for 30 sec - exiting 11:03:13 (5576): No heartbeat from core client for 30 sec - exiting 11:03:14 (5576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5936, iMonCtr=1 Model crash detected, will try to restart... 14:47:44 (3000): No heartbeat from core client for 30 sec - exiting 14:47:46 (3000): No heartbeat from core client for 30 sec - exiting 14:47:47 (3000): No heartbeat from core client for 30 sec - exiting 14:47:48 (3000): No heartbeat from core client for 30 sec - exiting 14:47:49 (3000): No heartbeat from core client for 30 sec - exiting 14:47:50 (3000): No heartbeat from core client for 30 sec - exiting 14:47:51 (3000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:30:19 (4060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:31:01 (6756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:48:57 (6892): No heartbeat from core client for 30 sec - exiting 14:48:58 (6892): No heartbeat from core client for 30 sec - exiting 14:48:59 (6892): No heartbeat from core client for 30 sec - exiting 14:49:01 (6892): No heartbeat from core client for 30 sec - exiting 14:49:02 (6892): No heartbeat from core client for 30 sec - exiting 14:49:03 (6892): No heartbeat from core client for 30 sec - exiting 14:49:04 (6892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:12:53 (6452): No heartbeat from core client for 30 sec - exiting 16:12:54 (6452): No heartbeat from core client for 30 sec - exiting 16:12:55 (6452): No heartbeat from core client for 30 sec - exiting 16:12:57 (6452): No heartbeat from core client for 30 sec - exiting 16:12:58 (6452): No heartbeat from core client for 30 sec - exiting 16:12:59 (6452): No heartbeat from core client for 30 sec - exiting 16:13:00 (6452): No heartbeat from core client for 30 sec - exiting 16:13:01 (6452): No heartbeat from core client for 30 sec - exiting 16:13:02 (6452): No heartbeat from core client for 30 sec - exiting 16:13:03 (6452): No heartbeat from core client for 30 sec - exiting 16:13:04 (6452): No heartbeat from core client for 30 sec - exiting 16:13:06 (6452): No heartbeat from core client for 30 sec - exiting 16:13:07 (6452): No heartbeat from core client for 30 sec - exiting 16:13:08 (6452): No heartbeat from core client for 30 sec - exiting 16:13:09 (6452): No heartbeat from core client for 30 sec - exiting 16:13:10 (6452): No heartbeat from core client for 30 sec - exiting 16:13:11 (6452): No heartbeat from core client for 30 sec - exiting 16:13:12 (6452): No heartbeat from core client for 30 sec - exiting 16:13:13 (6452): No heartbeat from core client for 30 sec - exiting 16:13:14 (6452): No heartbeat from core client for 30 sec - exiting 16:13:15 (6452): No heartbeat from core client for 30 sec - exiting 16:13:16 (6452): No heartbeat from core client for 30 sec - exiting 16:13:18 (6452): No heartbeat from core client for 30 sec - exiting 16:13:19 (6452): No heartbeat from core client for 30 sec - exiting 16:13:20 (6452): No heartbeat from core client for 30 sec - exiting 16:13:21 (6452): No heartbeat from core client for 30 sec - exiting 16:13:22 (6452): No heartbeat from core client for 30 sec - exiting 16:13:23 (6452): No heartbeat from core client for 30 sec - exiting 16:13:24 (6452): No heartbeat from core client for 30 sec - exiting 16:13:25 (6452): No heartbeat from core client for 30 sec - exiting 16:13:26 (6452): No heartbeat from core client for 30 sec - exiting 16:13:27 (6452): No heartbeat from core client for 30 sec - exiting 16:13:28 (6452): No heartbeat from core client for 30 sec - exiting 16:13:30 (6452): No heartbeat from core client for 30 sec - exiting 16:13:31 (6452): No heartbeat from core client for 30 sec - exiting 16:13:32 (6452): No heartbeat from core client for 30 sec - exiting 16:13:33 (6452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:35:06 (7136): No heartbeat from core client for 30 sec - exiting 18:35:07 (7136): No heartbeat from core client for 30 sec - exiting 18:35:08 (7136): No heartbeat from core client for 30 sec - exiting 18:35:10 (7136): No heartbeat from core client for 30 sec - exiting 18:35:11 (7136): No heartbeat from core client for 30 sec - exiting 18:35:12 (7136): No heartbeat from core client for 30 sec - exiting 18:35:13 (7136): No heartbeat from core client for 30 sec - exiting 18:35:14 (7136): No heartbeat from core client for 30 sec - exiting 18:35:15 (7136): No heartbeat from core client for 30 sec - exiting 18:35:16 (7136): No heartbeat from core client for 30 sec - exiting 18:35:17 (7136): No heartbeat from core client for 30 sec - exiting 18:35:18 (7136): No heartbeat from core client for 30 sec - exiting 18:35:19 (7136): No heartbeat from core client for 30 sec - exiting 18:35:20 (7136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:50:02 (4972): No heartbeat from core client for 30 sec - exiting 19:50:04 (4972): No heartbeat from core client for 30 sec - exiting 19:50:05 (4972): No heartbeat from core client for 30 sec - exiting 19:50:06 (4972): No heartbeat from core client for 30 sec - exiting 19:50:07 (4972): No heartbeat from core client for 30 sec - exiting 19:50:08 (4972): No heartbeat from core client for 30 sec - exiting 19:50:09 (4972): No heartbeat from core client for 30 sec - exiting 19:50:10 (4972): No heartbeat from core client for 30 sec - exiting 19:50:12 (4972): No heartbeat from core client for 30 sec - exiting 19:50:13 (4972): No heartbeat from core client for 30 sec - exiting 19:50:15 (4972): No heartbeat from core client for 30 sec - exiting 19:50:16 (4972): No heartbeat from core client for 30 sec - exiting 19:50:17 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:50:18 (4972): No heartbeat from core client for 30 sec - exiting 20:56:18 (5344): No heartbeat from core client for 30 sec - exiting 20:56:19 (5344): No heartbeat from core client for 30 sec - exiting 20:56:20 (5344): No heartbeat from core client for 30 sec - exiting 20:56:21 (5344): No heartbeat from core client for 30 sec - exiting 20:56:23 (5344): No heartbeat from core client for 30 sec - exiting 20:56:24 (5344): No heartbeat from core client for 30 sec - exiting 20:56:25 (5344): No heartbeat from core client for 30 sec - exiting 20:56:26 (5344): No heartbeat from core client for 30 sec - exiting 20:56:27 (5344): No heartbeat from core client for 30 sec - exiting 20:56:28 (5344): No heartbeat from core client for 30 sec - exiting 20:56:29 (5344): No heartbeat from core client for 30 sec - exiting 20:56:30 (5344): No heartbeat from core client for 30 sec - exiting 20:56:31 (5344): No heartbeat from core client for 30 sec - exiting 20:56:32 (5344): No heartbeat from core client for 30 sec - exiting 20:56:33 (5344): No heartbeat from core client for 30 sec - exiting 20:56:35 (5344): No heartbeat from core client for 30 sec - exiting 20:56:36 (5344): No heartbeat from core client for 30 sec - exiting 20:56:37 (5344): No heartbeat from core client for 30 sec - exiting 20:56:38 (5344): No heartbeat from core client for 30 sec - exiting 20:56:39 (5344): No heartbeat from core client for 30 sec - exiting 20:56:40 (5344): No heartbeat from core client for 30 sec - exiting 20:56:41 (5344): No heartbeat from core client for 30 sec - exiting 20:56:42 (5344): No heartbeat from core client for 30 sec - exiting 20:56:43 (5344): No heartbeat from core client for 30 sec - exiting 20:56:44 (5344): No heartbeat from core client for 30 sec - exiting 20:56:45 (5344): No heartbeat from core client for 30 sec - exiting 20:56:47 (5344): No heartbeat from core client for 30 sec - exiting 20:56:48 (5344): No heartbeat from core client for 30 sec - exiting 20:56:49 (5344): No heartbeat from core client for 30 sec - exiting 20:56:50 (5344): No heartbeat from core client for 30 sec - exiting 20:56:51 (5344): No heartbeat from core client for 30 sec - exiting 20:56:52 (5344): No heartbeat from core client for 30 sec - exiting 20:56:53 (5344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:14:43 (1480): No heartbeat from core client for 30 sec - exiting 19:14:44 (1480): No heartbeat from core client for 30 sec - exiting 19:14:45 (1480): No heartbeat from core client for 30 sec - exiting 19:14:46 (1480): No heartbeat from core client for 30 sec - exiting 19:14:47 (1480): No heartbeat from core client for 30 sec - exiting 19:14:49 (1480): No heartbeat from core client for 30 sec - exiting 19:14:50 (1480): No heartbeat from core client for 30 sec - exiting 19:14:51 (1480): No heartbeat from core client for 30 sec - exiting 19:14:52 (1480): No heartbeat from core client for 30 sec - exiting 19:14:53 (1480): No heartbeat from core client for 30 sec - exiting 19:14:54 (1480): No heartbeat from core client for 30 sec - exiting 19:14:55 (1480): No heartbeat from core client for 30 sec - exiting 19:14:56 (1480): No heartbeat from core client for 30 sec - exiting 19:14:57 (1480): No heartbeat from core client for 30 sec - exiting 19:14:58 (1480): No heartbeat from core client for 30 sec - exiting 19:14:59 (1480): No heartbeat from core client for 30 sec - exiting 19:15:00 (1480): No heartbeat from core client for 30 sec - exiting 19:15:02 (1480): No heartbeat from core client for 30 sec - exiting 19:15:03 (1480): No heartbeat from core client for 30 sec - exiting 19:15:04 (1480): No heartbeat from core client for 30 sec - exiting 19:15:05 (1480): No heartbeat from core client for 30 sec - exiting 19:15:06 (1480): No heartbeat from core client for 30 sec - exiting 19:15:07 (1480): No heartbeat from core client for 30 sec - exiting 19:15:08 (1480): No heartbeat from core client for 30 sec - exiting 19:15:09 (1480): No heartbeat from core client for 30 sec - exiting 19:15:10 (1480): No heartbeat from core client for 30 sec - exiting 19:15:11 (1480): No heartbeat from core client for 30 sec - exiting 19:15:12 (1480): No heartbeat from core client for 30 sec - exiting 19:15:14 (1480): No heartbeat from core client for 30 sec - exiting 19:15:15 (1480): No heartbeat from core client for 30 sec - exiting 19:15:16 (1480): No heartbeat from core client for 30 sec - exiting 19:15:17 (1480): No heartbeat from core client for 30 sec - exiting 19:15:18 (1480): No heartbeat from core client for 30 sec - exiting 19:15:19 (1480): No heartbeat from core client for 30 sec - exiting 19:15:20 (1480): No heartbeat from core client for 30 sec - exiting 19:15:21 (1480): No heartbeat from core client for 30 sec - exiting 19:15:22 (1480): No heartbeat from core client for 30 sec - exiting 19:15:23 (1480): No heartbeat from core client for 30 sec - exiting 19:15:24 (1480): No heartbeat from core client for 30 sec - exiting 19:15:26 (1480): No heartbeat from core client for 30 sec - exiting 19:15:27 (1480): No heartbeat from core client for 30 sec - exiting 19:15:28 (1480): No heartbeat from core client for 30 sec - exiting 19:15:29 (1480): No heartbeat from core client for 30 sec - exiting 19:15:30 (1480): No heartbeat from core client for 30 sec - exiting 19:15:31 (1480): No heartbeat from core client for 30 sec - exiting 19:15:32 (1480): No heartbeat from core client for 30 sec - exiting 19:15:33 (1480): No heartbeat from core client for 30 sec - exiting 19:15:34 (1480): No heartbeat from core client for 30 sec - exiting 19:15:35 (1480): No heartbeat from core client for 30 sec - exiting 19:15:37 (1480): No heartbeat from core client for 30 sec - exiting 19:15:38 (1480): No heartbeat from core client for 30 sec - exiting 19:15:39 (1480): No heartbeat from core client for 30 sec - exiting 19:15:40 (1480): No heartbeat from core client for 30 sec - exiting 19:15:41 (1480): No heartbeat from core client for 30 sec - exiting 19:15:42 (1480): No heartbeat from core client for 30 sec - exiting 19:15:43 (1480): No heartbeat from core client for 30 sec - exiting 19:15:44 (1480): No heartbeat from core client for 30 sec - exiting 19:15:45 (1480): No heartbeat from core client for 30 sec - exiting 19:15:46 (1480): No heartbeat from core client for 30 sec - exiting 19:15:48 (1480): No heartbeat from core client for 30 sec - exiting 19:15:49 (1480): No heartbeat from core client for 30 sec - exiting 19:15:50 (1480): No heartbeat from core client for 30 sec - exiting 19:15:51 (1480): No heartbeat from core client for 30 sec - exiting 19:15:52 (1480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:00:04 (6228): No heartbeat from core client for 30 sec - exiting 08:00:05 (6228): No heartbeat from core client for 30 sec - exiting 08:00:06 (6228): No heartbeat from core client for 30 sec - exiting 08:00:07 (6228): No heartbeat from core client for 30 sec - exiting 08:00:08 (6228): No heartbeat from core client for 30 sec - exiting 08:00:09 (6228): No heartbeat from core client for 30 sec - exiting 08:00:11 (6228): No heartbeat from core client for 30 sec - exiting 08:00:12 (6228): No heartbeat from core client for 30 sec - exiting 08:00:13 (6228): No heartbeat from core client for 30 sec - exiting 08:00:14 (6228): No heartbeat from core client for 30 sec - exiting 08:00:15 (6228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:20:51 (3508): No heartbeat from core client for 30 sec - exiting 20:20:52 (3508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6132, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5216, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5792, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 21:18:07 (5796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... 14:26:24 (5412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied 19:18:20 (2328): No heartbeat from core client for 30 sec - exiting 19:18:21 (2328): No heartbeat from core client for 30 sec - exiting 19:18:22 (2328): No heartbeat from core client for 30 sec - exiting 19:18:24 (2328): No heartbeat from core client for 30 sec - exiting 19:18:25 (2328): No heartbeat from core client for 30 sec - exiting 19:18:26 (2328): No heartbeat from core client for 30 sec - exiting 19:18:27 (2328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:50:03 (5644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:32:17 (2004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:32 (2228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:59:49 (6468): No heartbeat from core client for 30 sec - exiting 20:59:50 (6468): No heartbeat from core client for 30 sec - exiting 20:59:51 (6468): No heartbeat from core client for 30 sec - exiting 20:59:52 (6468): No heartbeat from core client for 30 sec - exiting 20:59:53 (6468): No heartbeat from core client for 30 sec - exiting 20:59:54 (6468): No heartbeat from core client for 30 sec - exiting 20:59:55 (6468): No heartbeat from core client for 30 sec - exiting 20:59:56 (6468): No heartbeat from core client for 30 sec - exiting 20:59:57 (6468): No heartbeat from core client for 30 sec - exiting 20:59:58 (6468): No heartbeat from core client for 30 sec - exiting 21:00:00 (6468): No heartbeat from core client for 30 sec - exiting 21:00:01 (6468): No heartbeat from core client for 30 sec - exiting 21:00:02 (6468): No heartbeat from core client for 30 sec - exiting 21:00:03 (6468): No heartbeat from core client for 30 sec - exiting 21:00:04 (6468): No heartbeat from core client for 30 sec - exiting 21:00:05 (6468): No heartbeat from core client for 30 sec - exiting 21:00:06 (6468): No heartbeat from core client for 30 sec - exiting 21:00:07 (6468): No heartbeat from core client for 30 sec - exiting 21:00:08 (6468): No heartbeat from core client for 30 sec - exiting 21:00:10 (6468): No heartbeat from core client for 30 sec - exiting 21:00:11 (6468): No heartbeat from core client for 30 sec - exiting 21:00:12 (6468): No heartbeat from core client for 30 sec - exiting 21:00:13 (6468): No heartbeat from core client for 30 sec - exiting 21:00:14 (6468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:04 (3688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:35:01 (6916): No heartbeat from core client for 30 sec - exiting 15:35:03 (6916): No heartbeat from core client for 30 sec - exiting 15:35:04 (6916): No heartbeat from core client for 30 sec - exiting 15:35:05 (6916): No heartbeat from core client for 30 sec - exiting 15:35:06 (6916): No heartbeat from core client for 30 sec - exiting 15:35:07 (6916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=772, iMonCtr=1 Model crash detected, will try to restart... 09:10:11 (6876): No heartbeat from core client for 30 sec - exiting 09:10:12 (6876): No heartbeat from core client for 30 sec - exiting 09:10:13 (6876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied 20:52:51 (5496): No heartbeat from core client for 30 sec - exiting 20:52:52 (5496): No heartbeat from core client for 30 sec - exiting 20:52:53 (5496): No heartbeat from core client for 30 sec - exiting 20:52:54 (5496): No heartbeat from core client for 30 sec - exiting 20:52:55 (5496): No heartbeat from core client for 30 sec - exiting 20:52:56 (5496): No heartbeat from core client for 30 sec - exiting 20:52:57 (5496): No heartbeat from core client for 30 sec - exiting 20:52:59 (5496): No heartbeat from core client for 30 sec - exiting 20:53:00 (5496): No heartbeat from core client for 30 sec - exiting 20:53:01 (5496): No heartbeat from core client for 30 sec - exiting 20:53:02 (5496): No heartbeat from core client for 30 sec - exiting 20:53:03 (5496): No heartbeat from core client for 30 sec - exiting 20:53:04 (5496): No heartbeat from core client for 30 sec - exiting 20:53:05 (5496): No heartbeat from core client for 30 sec - exiting 20:53:06 (5496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:28:09 (6304): No heartbeat from core client for 30 sec - exiting 21:28:10 (6304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:43:35 (5692): No heartbeat from core client for 30 sec - exiting 20:43:36 (5692): No heartbeat from core client for 30 sec - exiting 20:43:37 (5692): No heartbeat from core client for 30 sec - exiting 20:43:39 (5692): No heartbeat from core client for 30 sec - exiting 20:43:40 (5692): No heartbeat from core client for 30 sec - exiting 20:43:41 (5692): No heartbeat from core client for 30 sec - exiting 20:43:42 (5692): No heartbeat from core client for 30 sec - exiting 20:43:43 (5692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:57:24 (4108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:20 (6772): No heartbeat from core client for 30 sec - exiting 18:13:21 (6772): No heartbeat from core client for 30 sec - exiting 18:13:22 (6772): No heartbeat from core client for 30 sec - exiting 18:13:23 (6772): No heartbeat from core client for 30 sec - exiting 18:13:24 (6772): No heartbeat from core client for 30 sec - exiting 18:13:25 (6772): No heartbeat from core client for 30 sec - exiting 18:13:26 (6772): No heartbeat from core client for 30 sec - exiting 18:13:27 (6772): No heartbeat from core client for 30 sec - exiting 18:13:28 (6772): No heartbeat from core client for 30 sec - exiting 18:13:29 (6772): No heartbeat from core client for 30 sec - exiting 18:13:31 (6772): No heartbeat from core client for 30 sec - exiting 18:13:32 (6772): No heartbeat from core client for 30 sec - exiting 18:13:33 (6772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... 13:51:35 (4364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found 14:06:05 (664): No heartbeat from core client for 30 sec - exiting 14:06:07 (664): No heartbeat from core client for 30 sec - exiting 14:06:08 (664): No heartbeat from core client for 30 sec - exiting 14:06:09 (664): No heartbeat from core client for 30 sec - exiting 14:06:10 (664): No heartbeat from core client for 30 sec - exiting 14:06:11 (664): No heartbeat from core client for 30 sec - exiting 14:06:12 (664): No heartbeat from core client for 30 sec - exiting 14:06:13 (664): No heartbeat from core client for 30 sec - exiting 14:06:14 (664): No heartbeat from core client for 30 sec - exiting 14:06:15 (664): No heartbeat from core client for 30 sec - exiting 14:06:16 (664): No heartbeat from core client for 30 sec - exiting 14:06:18 (664): No heartbeat from core client for 30 sec - exiting 14:06:19 (664): No heartbeat from core client for 30 sec - exiting 14:06:20 (664): No heartbeat from core client for 30 sec - exiting 14:06:21 (664): No heartbeat from core client for 30 sec - exiting 14:06:22 (664): No heartbeat from core client for 30 sec - exiting 14:06:23 (664): No heartbeat from core client for 30 sec - exiting 14:06:24 (664): No heartbeat from core client for 30 sec - exiting 14:06:25 (664): No heartbeat from core client for 30 sec - exiting 14:06:26 (664): No heartbeat from core client for 30 sec - exiting 14:06:27 (664): No heartbeat from core client for 30 sec - exiting 14:06:28 (664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5932, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x778C3AB3 read attempt to address 0x40C80860 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77CE3AB3 read attempt to address 0x40C80860 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3bqy_1980_40_008320699/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Jun 2013 10:28:58	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	1,036,800	1,363,868	1.3155
09 Jun 2013 04:28:02	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	1,010,880	1,341,996	1.3276
08 Jun 2013 11:19:42	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	984,960	1,320,928	1.3411
06 Jun 2013 11:32:40	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	959,040	1,300,322	1.3559
05 Jun 2013 11:26:29	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	933,120	1,279,577	1.3713
03 Jun 2013 11:45:58	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	907,200	1,256,917	1.3855
01 Jun 2013 12:10:41	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	881,280	1,229,081	1.3947
01 Jun 2013 00:47:17	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	855,360	1,195,738	1.3979
27 May 2013 13:40:40	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	829,440	1,157,260	1.3952
26 May 2013 04:59:03	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	803,520	1,119,151	1.3928
23 May 2013 12:24:18	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	777,600	1,078,842	1.3874
20 May 2013 13:27:33	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	751,680	1,040,232	1.3839
18 May 2013 22:53:25	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	725,760	1,002,117	1.3808
15 May 2013 11:01:18	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	699,840	963,335	1.3765
12 May 2013 02:18:40	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	673,920	923,663	1.3706
10 May 2013 11:07:09	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	648,000	882,706	1.3622
07 May 2013 10:17:03	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	622,080	838,055	1.3472
06 May 2013 04:32:22	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	596,160	793,915	1.3317
04 May 2013 00:08:50	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	570,240	748,907	1.3133
03 May 2013 00:31:13	1277214	15701858	hadcm3n_3bqy_1980_40_008320699_2	544,320	704,542	1.2944