Task 15821226

Name	hadcm3n_o4lq_1980_40_008385823_0
Workunit	8536682
Created	3 Jun 2013, 4:28:02 UTC
Sent	9 Jun 2013, 8:59:35 UTC
Report deadline	8 Sep 2013, 16:26:46 UTC
Received	9 Sep 2013, 3:39:33 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1166408
Run time	10 days 15 hours 32 min 26 sec
CPU time	9 days 20 hours 33 min 5 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.18 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 16:36:22 (2956): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 17:38:31 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:07:15 (7044): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 12:07:18 (7044): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:27:51 (4716): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 06:27:52 (4716): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5476, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:44:03 (1804): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 19:44:04 (1804): No heartbeat from core client for 30 sec - exiting 19:44:05 (1804): No heartbeat from core client for 30 sec - exiting 19:44:06 (1804): No heartbeat from core client for 30 sec - exiting 19:44:07 (1804): No heartbeat from core client for 30 sec - exiting 19:44:08 (1804): No heartbeat from core client for 30 sec - exiting 19:44:09 (1804): No heartbeat from core client for 30 sec - exiting 19:44:10 (1804): No heartbeat from core client for 30 sec - exiting 19:44:11 (1804): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:09:26 (4852): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 18:09:28 (4852): No heartbeat from core client for 30 sec - exiting 18:09:29 (4852): No heartbeat from core client for 30 sec - exiting 18:09:30 (4852): No heartbeat from core client for 30 sec - exiting 18:09:31 (4852): No heartbeat from core client for 30 sec - exiting 18:09:32 (4852): No heartbeat from core client for 30 sec - exiting 18:09:33 (4852): No heartbeat from core client for 30 sec - exiting 18:09:34 (4852): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:20:39 (2820): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 12:20:41 (2820): No heartbeat from core client for 30 sec - exiting 12:20:42 (2820): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 21:21:47 (3532): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 21:21:50 (3532): No heartbeat from core client for 30 sec - exiting 21:21:51 (3532): No heartbeat from core client for 30 sec - exiting 21:21:52 (3532): No heartbeat from core client for 30 sec - exiting 21:21:53 (3532): No heartbeat from core client for 30 sec - exiting 21:21:54 (3532): No heartbeat from core client for 30 sec - exiting 21:21:55 (3532): No heartbeat from core client for 30 sec - exiting 21:21:56 (3532): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:04:30 (3036): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 08:04:34 (3036): No heartbeat from core client for 30 sec - exiting 08:04:35 (3036): No heartbeat from core client for 30 sec - exiting 08:04:36 (3036): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:20:47 (5604): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 09:20:50 (5604): No heartbeat from core client for 30 sec - exiting 09:20:51 (5604): No heartbeat from core client for 30 sec - exiting 09:20:52 (5604): No heartbeat from core client for 30 sec - exiting 09:20:53 (5604): No heartbeat from core client for 30 sec - exiting 09:20:54 (5604): No heartbeat from core client for 30 sec - exiting 09:20:55 (5604): No heartbeat from core client for 30 sec - exiting 09:20:56 (5604): No heartbeat from core client for 30 sec - exiting 09:20:57 (5604): No heartbeat from core client for 30 sec - exiting 09:20:58 (5604): No heartbeat from core client for 30 sec - exiting 09:20:59 (5604): No heartbeat from core client for 30 sec - exiting 09:21:00 (5604): No heartbeat from core client for 30 sec - exiting 09:21:01 (5604): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:16:52 (4312): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 09:16:56 (4312): No heartbeat from core client for 30 sec - exiting 09:16:57 (4312): No heartbeat from core client for 30 sec - exiting 09:16:58 (4312): No heartbeat from core client for 30 sec - exiting 09:16:59 (4312): No heartbeat from core client for 30 sec - exiting 09:17:00 (4312): No heartbeat from core client for 30 sec - exiting 09:17:01 (4312): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:41:30 (756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:28:13 (3936): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 22:28:17 (3936): No heartbeat from core client for 30 sec - exiting 22:28:18 (3936): No heartbeat from core client for 30 sec - exiting 22:28:19 (3936): No heartbeat from core client for 30 sec - exiting 22:28:20 (3936): No heartbeat from core client for 30 sec - exiting 22:28:21 (3936): No heartbeat from core client for 30 sec - exiting 17:16:00 (5208): No heartbeat from core client for 30 sec - exiting 17:16:02 (5208): No heartbeat from core client for 30 sec - exiting 17:16:03 (5208): No heartbeat from core client for 30 sec - exiting 17:16:04 (5208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:30 (888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:32 (888): No heartbeat from core client for 30 sec - exiting 17:30:33 (888): No heartbeat from core client for 30 sec - exiting 17:30:34 (888): No heartbeat from core client for 30 sec - exiting 17:30:35 (888): No heartbeat from core client for 30 sec - exiting 17:30:36 (888): No heartbeat from core client for 30 sec - exiting 17:34:03 (4172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:34:08 (4172): No heartbeat from core client for 30 sec - exiting 17:34:09 (4172): No heartbeat from core client for 30 sec - exiting 17:34:11 (4172): No heartbeat from core client for 30 sec - exiting 17:34:12 (4172): No heartbeat from core client for 30 sec - exiting 17:34:13 (4172): No heartbeat from core client for 30 sec - exiting 17:34:14 (4172): No heartbeat from core client for 30 sec - exiting 17:34:15 (4172): No heartbeat from core client for 30 sec - exiting 17:34:16 (4172): No heartbeat from core client for 30 sec - exiting 17:34:17 (4172): No heartbeat from core client for 30 sec - exiting 17:34:18 (4172): No heartbeat from core client for 30 sec - exiting 17:34:19 (4172): No heartbeat from core client for 30 sec - exiting 17:34:20 (4172): No heartbeat from core client for 30 sec - exiting 17:34:21 (4172): No heartbeat from core client for 30 sec - exiting 17:34:22 (4172): No heartbeat from core client for 30 sec - exiting 17:36:02 (4360): No heartbeat from core client for 30 sec - exiting 17:36:03 (4360): No heartbeat from core client for 30 sec - exiting 17:36:04 (4360): No heartbeat from core client for 30 sec - exiting 17:36:05 (4360): No heartbeat from core client for 30 sec - exiting 17:36:06 (4360): No heartbeat from core client for 30 sec - exiting 17:36:07 (4360): No heartbeat from core client for 30 sec - exiting 17:36:08 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:37:58 (4140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:39:54 (4112): No heartbeat from core client for 30 sec - exiting 17:39:55 (4112): No heartbeat from core client for 30 sec - exiting 17:39:56 (4112): No heartbeat from core client for 30 sec - exiting 17:39:57 (4112): No heartbeat from core client for 30 sec - exiting 17:39:58 (4112): No heartbeat from core client for 30 sec - exiting 17:39:59 (4112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:47:29 (4552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:47:32 (4552): No heartbeat from core client for 30 sec - exiting 17:47:33 (4552): No heartbeat from core client for 30 sec - exiting 17:47:34 (4552): No heartbeat from core client for 30 sec - exiting 17:47:35 (4552): No heartbeat from core client for 30 sec - exiting 17:47:36 (4552): No heartbeat from core client for 30 sec - exiting 17:50:53 (680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:50:55 (680): No heartbeat from core client for 30 sec - exiting 17:50:56 (680): No heartbeat from core client for 30 sec - exiting 17:50:58 (680): No heartbeat from core client for 30 sec - exiting 17:50:59 (680): No heartbeat from core client for 30 sec - exiting 17:51:00 (680): No heartbeat from core client for 30 sec - exiting 17:51:01 (680): No heartbeat from core client for 30 sec - exiting 17:51:02 (680): No heartbeat from core client for 30 sec - exiting 17:51:03 (680): No heartbeat from core client for 30 sec - exiting 17:51:04 (680): No heartbeat from core client for 30 sec - exiting 17:51:05 (680): No heartbeat from core client for 30 sec - exiting 17:51:06 (680): No heartbeat from core client for 30 sec - exiting 17:51:07 (680): No heartbeat from core client for 30 sec - exiting 17:51:08 (680): No heartbeat from core client for 30 sec - exiting 17:51:09 (680): No heartbeat from core client for 30 sec - exiting 17:51:10 (680): No heartbeat from core client for 30 sec - exiting 17:51:11 (680): No heartbeat from core client for 30 sec - exiting 17:52:17 (3756): No heartbeat from core client for 30 sec - exiting 17:52:18 (3756): No heartbeat from core client for 30 sec - exiting 17:52:19 (3756): No heartbeat from core client for 30 sec - exiting 17:52:20 (3756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:54:17 (5624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:54:22 (5624): No heartbeat from core client for 30 sec - exiting 17:54:24 (5624): No heartbeat from core client for 30 sec - exiting 17:54:25 (5624): No heartbeat from core client for 30 sec - exiting 17:54:26 (5624): No heartbeat from core client for 30 sec - exiting 17:54:27 (5624): No heartbeat from core client for 30 sec - exiting 17:54:28 (5624): No heartbeat from core client for 30 sec - exiting 17:54:29 (5624): No heartbeat from core client for 30 sec - exiting 17:54:30 (5624): No heartbeat from core client for 30 sec - exiting 17:54:31 (5624): No heartbeat from core client for 30 sec - exiting 17:54:32 (5624): No heartbeat from core client for 30 sec - exiting 17:54:33 (5624): No heartbeat from core client for 30 sec - exiting 17:54:34 (5624): No heartbeat from core client for 30 sec - exiting 17:54:35 (5624): No heartbeat from core client for 30 sec - exiting 17:54:37 (5624): No heartbeat from core client for 30 sec - exiting 17:54:38 (5624): No heartbeat from core client for 30 sec - exiting 17:54:39 (5624): No heartbeat from core client for 30 sec - exiting 17:54:40 (5624): No heartbeat from core client for 30 sec - exiting 17:56:10 (760): No heartbeat from core client for 30 sec - exiting 17:56:11 (760): No heartbeat from core client for 30 sec - exiting 17:56:12 (760): No heartbeat from core client for 30 sec - exiting 17:56:13 (760): No heartbeat from core client for 30 sec - exiting 17:56:14 (760): No heartbeat from core client for 30 sec - exiting 17:56:15 (760): No heartbeat from core client for 30 sec - exiting 17:56:16 (760): No heartbeat from core client for 30 sec - exiting 17:56:17 (760): No heartbeat from core client for 30 sec - exiting 17:56:18 (760): No heartbeat from core client for 30 sec - exiting 17:56:19 (760): No heartbeat from core client for 30 sec - exiting 17:56:20 (760): No heartbeat from core client for 30 sec - exiting 17:56:21 (760): No heartbeat from core client for 30 sec - exiting 17:56:22 (760): No heartbeat from core client for 30 sec - exiting 17:56:23 (760): No heartbeat from core client for 30 sec - exiting 17:56:24 (760): No heartbeat from core client for 30 sec - exiting 17:56:25 (760): No heartbeat from core client for 30 sec - exiting 17:56:26 (760): No heartbeat from core client for 30 sec - exiting 17:56:27 (760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:19 (5048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:26 (5048): No heartbeat from core client for 30 sec - exiting 17:58:28 (5048): No heartbeat from core client for 30 sec - exiting 17:58:29 (5048): No heartbeat from core client for 30 sec - exiting 17:59:53 (5168): No heartbeat from core client for 30 sec - exiting 17:59:54 (5168): No heartbeat from core client for 30 sec - exiting 17:59:55 (5168): No heartbeat from core client for 30 sec - exiting 17:59:56 (5168): No heartbeat from core client for 30 sec - exiting 17:59:57 (5168): No heartbeat from core client for 30 sec - exiting 17:59:58 (5168): No heartbeat from core client for 30 sec - exiting 17:59:59 (5168): No heartbeat from core client for 30 sec - exiting 18:00:00 (5168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:00:01 (5168): No heartbeat from core client for 30 sec - exiting 18:01:56 (3548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:01 (3548): No heartbeat from core client for 30 sec - exiting 18:02:02 (3548): No heartbeat from core client for 30 sec - exiting 18:02:04 (3548): No heartbeat from core client for 30 sec - exiting 18:02:05 (3548): No heartbeat from core client for 30 sec - exiting 18:02:06 (3548): No heartbeat from core client for 30 sec - exiting 18:02:07 (3548): No heartbeat from core client for 30 sec - exiting 18:02:08 (3548): No heartbeat from core client for 30 sec - exiting 18:02:09 (3548): No heartbeat from core client for 30 sec - exiting 18:02:10 (3548): No heartbeat from core client for 30 sec - exiting 18:02:11 (3548): No heartbeat from core client for 30 sec - exiting 18:02:12 (3548): No heartbeat from core client for 30 sec - exiting 18:02:13 (3548): No heartbeat from core client for 30 sec - exiting 18:02:14 (3548): No heartbeat from core client for 30 sec - exiting 18:13:50 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:51 (4684): No heartbeat from core client for 30 sec - exiting 18:13:52 (4684): No heartbeat from core client for 30 sec - exiting 18:13:53 (4684): No heartbeat from core client for 30 sec - exiting 18:15:14 (5328): No heartbeat from core client for 30 sec - exiting 18:15:15 (5328): No heartbeat from core client for 30 sec - exiting 18:15:16 (5328): No heartbeat from core client for 30 sec - exiting 18:15:17 (5328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:37:14 (4448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:37:16 (4448): No heartbeat from core client for 30 sec - exiting 18:37:17 (4448): No heartbeat from core client for 30 sec - exiting 18:37:18 (4448): No heartbeat from core client for 30 sec - exiting 18:37:19 (4448): No heartbeat from core client for 30 sec - exiting 18:37:20 (4448): No heartbeat from core client for 30 sec - exiting 18:37:21 (4448): No heartbeat from core client for 30 sec - exiting 18:37:22 (4448): No heartbeat from core client for 30 sec - exiting 18:37:23 (4448): No heartbeat from core client for 30 sec - exiting 18:40:33 (3888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:41:27 (5044): No heartbeat from core client for 30 sec - exiting 18:41:28 (5044): No heartbeat from core client for 30 sec - exiting 18:41:29 (5044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:23 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:31 (4904): No heartbeat from core client for 30 sec - exiting 18:45:27 (5160): No heartbeat from core client for 30 sec - exiting 18:45:28 (5160): No heartbeat from core client for 30 sec - exiting 18:45:29 (5160): No heartbeat from core client for 30 sec - exiting 18:45:30 (5160): No heartbeat from core client for 30 sec - exiting 18:45:31 (5160): No heartbeat from core client for 30 sec - exiting 18:45:32 (5160): No heartbeat from core client for 30 sec - exiting 18:45:33 (5160): No heartbeat from core client for 30 sec - exiting 18:45:34 (5160): No heartbeat from core client for 30 sec - exiting 18:45:35 (5160): No heartbeat from core client for 30 sec - exiting 18:45:36 (5160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:47:26 (6092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:47:32 (6092): No heartbeat from core client for 30 sec - exiting 18:47:34 (6092): No heartbeat from core client for 30 sec - exiting 18:47:35 (6092): No heartbeat from core client for 30 sec - exiting 18:47:36 (6092): No heartbeat from core client for 30 sec - exiting 18:49:35 (3004): No heartbeat from core client for 30 sec - exiting 18:49:36 (3004): No heartbeat from core client for 30 sec - exiting 18:49:37 (3004): No heartbeat from core client for 30 sec - exiting 18:49:38 (3004): No heartbeat from core client for 30 sec - exiting 18:49:39 (3004): No heartbeat from core client for 30 sec - exiting 18:49:40 (3004): No heartbeat from core client for 30 sec - exiting 18:49:41 (3004): No heartbeat from core client for 30 sec - exiting 18:49:42 (3004): No heartbeat from core client for 30 sec - exiting 18:49:43 (3004): No heartbeat from core client for 30 sec - exiting 18:49:44 (3004): No heartbeat from core client for 30 sec - exiting 18:49:45 (3004): No heartbeat from core client for 30 sec - exiting 18:49:46 (3004): No heartbeat from core client for 30 sec - exiting 18:49:47 (3004): No heartbeat from core client for 30 sec - exiting 18:49:48 (3004): No heartbeat from core client for 30 sec - exiting 18:49:49 (3004): No heartbeat from core client for 30 sec - exiting 18:49:50 (3004): No heartbeat from core client for 30 sec - exiting 18:49:51 (3004): No heartbeat from core client for 30 sec - exiting 18:49:52 (3004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:51:54 (4992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:52:02 (4992): No heartbeat from core client for 30 sec - exiting 18:52:03 (4992): No heartbeat from core client for 30 sec - exiting 18:52:04 (4992): No heartbeat from core client for 30 sec - exiting 18:52:05 (4992): No heartbeat from core client for 30 sec - exiting 18:52:07 (4992): No heartbeat from core client for 30 sec - exiting 18:52:08 (4992): No heartbeat from core client for 30 sec - exiting 18:52:09 (4992): No heartbeat from core client for 30 sec - exiting 18:52:10 (4992): No heartbeat from core client for 30 sec - exiting 18:52:11 (4992): No heartbeat from core client for 30 sec - exiting 18:52:12 (4992): No heartbeat from core client for 30 sec - exiting 18:55:35 (3748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:50 (5600): No heartbeat from core client for 30 sec - exiting 18:56:51 (5600): No heartbeat from core client for 30 sec - exiting 18:56:52 (5600): No heartbeat from core client for 30 sec - exiting 18:56:53 (5600): No heartbeat from core client for 30 sec - exiting 18:56:54 (5600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:58:52 (2688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:58:57 (2688): No heartbeat from core client for 30 sec - exiting 18:58:58 (2688): No heartbeat from core client for 30 sec - exiting 18:58:59 (2688): No heartbeat from core client for 30 sec - exiting 18:59:00 (2688): No heartbeat from core client for 30 sec - exiting 18:59:01 (2688): No heartbeat from core client for 30 sec - exiting 18:59:02 (2688): No heartbeat from core client for 30 sec - exiting 18:59:03 (2688): No heartbeat from core client for 30 sec - exiting 18:59:04 (2688): No heartbeat from core client for 30 sec - exiting 18:59:05 (2688): No heartbeat from core client for 30 sec - exiting 18:59:06 (2688): No heartbeat from core client for 30 sec - exiting 18:59:07 (2688): No heartbeat from core client for 30 sec - exiting 18:59:08 (2688): No heartbeat from core client for 30 sec - exiting 18:59:09 (2688): No heartbeat from core client for 30 sec - exiting 18:59:10 (2688): No heartbeat from core client for 30 sec - exiting 18:59:11 (2688): No heartbeat from core client for 30 sec - exiting 18:59:12 (2688): No heartbeat from core client for 30 sec - exiting 18:59:13 (2688): No heartbeat from core client for 30 sec - exiting 18:59:14 (2688): No heartbeat from core client for 30 sec - exiting 19:01:41 (5924): No heartbeat from core client for 30 sec - exiting 19:01:42 (5924): No heartbeat from core client for 30 sec - exiting 19:01:43 (5924): No heartbeat from core client for 30 sec - exiting 19:01:44 (5924): No heartbeat from core client for 30 sec - exiting 19:01:45 (5924): No heartbeat from core client for 30 sec - exiting 19:01:46 (5924): No heartbeat from core client for 30 sec - exiting 19:01:47 (5924): No heartbeat from core client for 30 sec - exiting 19:01:48 (5924): No heartbeat from core client for 30 sec - exiting 19:01:49 (5924): No heartbeat from core client for 30 sec - exiting 19:01:50 (5924): No heartbeat from core client for 30 sec - exiting 19:01:51 (5924): No heartbeat from core client for 30 sec - exiting 19:01:52 (5924): No heartbeat from core client for 30 sec - exiting 19:01:53 (5924): No heartbeat from core client for 30 sec - exiting 19:01:54 (5924): No heartbeat from core client for 30 sec - exiting 19:01:55 (5924): No heartbeat from core client for 30 sec - exiting 19:01:56 (5924): No heartbeat from core client for 30 sec - exiting 19:01:57 (5924): No heartbeat from core client for 30 sec - exiting 19:01:58 (5924): No heartbeat from core client for 30 sec - exiting 19:01:59 (5924): No heartbeat from core client for 30 sec - exiting 19:02:00 (5924): No heartbeat from core client for 30 sec - exiting 19:02:01 (5924): No heartbeat from core client for 30 sec - exiting 19:02:02 (5924): No heartbeat from core client for 30 sec - exiting 19:02:03 (5924): No heartbeat from core client for 30 sec - exiting 19:02:04 (5924): No heartbeat from core client for 30 sec - exiting 19:02:05 (5924): No heartbeat from core client for 30 sec - exiting 19:02:06 (5924): No heartbeat from core client for 30 sec - exiting 19:02:07 (5924): No heartbeat from core client for 30 sec - exiting 19:02:08 (5924): No heartbeat from core client for 30 sec - exiting 19:02:09 (5924): No heartbeat from core client for 30 sec - exiting 19:02:10 (5924): No heartbeat from core client for 30 sec - exiting 19:02:11 (5924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:04:52 (5900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:04:54 (5900): No heartbeat from core client for 30 sec - exiting 19:04:55 (5900): No heartbeat from core client for 30 sec - exiting 19:04:56 (5900): No heartbeat from core client for 30 sec - exiting 19:04:57 (5900): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6128, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:57:23 (1940): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 08:57:26 (1940): No heartbeat from core client for 30 sec - exiting 08:57:27 (1940): No heartbeat from core client for 30 sec - exiting 08:57:28 (1940): No heartbeat from core client for 30 sec - exiting 08:57:30 (1940): No heartbeat from core client for 30 sec - exiting 08:57:31 (1940): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:08:32 (1968): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 07:08:33 (1968): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:58:24 (4124): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 04:58:25 (4124): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:01:16 (5816): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 03:01:19 (5816): No heartbeat from core client for 30 sec - exiting 03:01:20 (5816): No heartbeat from core client for 30 sec - exiting 03:01:21 (5816): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:20:42 (3000): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 13:20:43 (3000): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:01:36 (760): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 05:01:41 (760): No heartbeat from core client for 30 sec - exiting 05:01:42 (760): No heartbeat from core client for 30 sec - exiting 05:01:43 (760): No heartbeat from core client for 30 sec - exiting 05:01:44 (760): No heartbeat from core client for 30 sec - exiting 05:01:45 (760): No heartbeat from core client for 30 sec - exiting 05:01:46 (760): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:59:32 (820): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 23:59:39 (820): No heartbeat from core client for 30 sec - exiting 23:59:40 (820): No heartbeat from core client for 30 sec - exiting 23:59:41 (820): No heartbeat from core client for 30 sec - exiting 23:59:42 (820): No heartbeat from core client for 30 sec - exiting 23:59:43 (820): No heartbeat from core client for 30 sec - exiting 23:59:44 (820): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 17:38:38 (5728): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 17:38:39 (5728): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:02:33 (5268): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 14:02:34 (5268): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:01:36 (4064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:04:31 (5908): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 18:04:33 (5908): No heartbeat from core client for 30 sec - exiting 18:04:34 (5908): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... Error converting file to netcdf: dataout/o4lqko.pjl9c10 Error converting file to netcdf: dataout/o4lqko.pil9c10 Error converting file to netcdf: dataout/o4lqko.pfl9c10 Error converting file to netcdf: dataout/o4lqka.phl9c10 Error converting file to netcdf: dataout/o4lqka.pgl9c10 Error converting file to netcdf: dataout/o4lqka.pel9c10 Error converting file to netcdf: dataout/o4lqka.pdl9c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:22:07 (2096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:08 (2096): No heartbeat from core client for 30 sec - exiting 16:22:09 (2096): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 10:55:35 (848): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 10:55:37 (848): No heartbeat from core client for 30 sec - exiting 10:55:38 (848): No heartbeat from core client for 30 sec - exiting 10:55:39 (848): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77AE3AC3 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77037383 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77433AC3 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Sep 2013 13:15:12	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	1,036,800	851,294	0.8211
06 Sep 2013 08:27:02	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	1,010,880	830,042	0.8211
01 Sep 2013 13:00:54	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	984,960	809,693	0.8221
30 Aug 2013 18:31:55	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	959,040	788,895	0.8226
28 Aug 2013 18:10:49	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	933,120	768,114	0.8232
27 Aug 2013 11:27:04	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	907,200	746,807	0.8232
26 Aug 2013 10:29:47	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	881,280	726,213	0.8240
22 Aug 2013 10:14:56	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	855,360	704,491	0.8236
20 Aug 2013 07:32:53	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	829,440	683,097	0.8236
17 Aug 2013 07:49:25	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	803,520	662,681	0.8247
15 Aug 2013 15:43:20	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	777,600	640,794	0.8241
14 Aug 2013 18:00:11	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	751,680	618,648	0.8230
14 Aug 2013 18:00:11	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	725,760	596,736	0.8222
14 Aug 2013 18:00:11	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	699,840	572,747	0.8184
14 Aug 2013 18:00:11	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	673,920	551,816	0.8188
14 Aug 2013 18:00:11	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	648,000	529,679	0.8174
14 Aug 2013 18:00:11	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	622,080	508,482	0.8174
14 Aug 2013 18:00:11	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	596,160	487,539	0.8178
14 Aug 2013 18:00:11	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	570,240	467,470	0.8198
07 Jul 2013 15:47:25	1166408	15821226	hadcm3n_o4lq_1980_40_008385823_0	544,320	447,516	0.8222