Task 15929518

Name	hadcm3n_o721_1980_40_008407763_0
Workunit	8558619
Created	20 Aug 2013, 12:58:06 UTC
Sent	20 Aug 2013, 13:08:28 UTC
Report deadline	19 Nov 2013, 20:35:39 UTC
Received	15 Oct 2013, 15:51:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1280857
Run time	13 days 21 hours 28 min 35 sec
CPU time	10 days 3 hours 1 min 47 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.53 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4676, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5292, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2260, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:56:31 (4020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:27:10 (2128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 17:47:46 (4244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:01:56 (3904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5792, iMonCtr=1 Model crash detected, will try to restart... 11:21:17 (2776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5592, iMonCtr=1 Model crash detected, will try to restart... 11:29:15 (4240): No heartbeat from core client for 30 sec - exiting 11:29:18 (4240): No heartbeat from core client for 30 sec - exiting 11:29:19 (4240): No heartbeat from core client for 30 sec - exiting 11:29:20 (4240): No heartbeat from core client for 30 sec - exiting 11:29:21 (4240): No heartbeat from core client for 30 sec - exiting 11:29:22 (4240): No heartbeat from core client for 30 sec - exiting 11:29:23 (4240): No heartbeat from core client for 30 sec - exiting 11:29:25 (4240): No heartbeat from core client for 30 sec - exiting 11:29:26 (4240): No heartbeat from core client for 30 sec - exiting 11:29:29 (4240): No heartbeat from core client for 30 sec - exiting 11:29:30 (4240): No heartbeat from core client for 30 sec - exiting 11:29:31 (4240): No heartbeat from core client for 30 sec - exiting 11:29:32 (4240): No heartbeat from core client for 30 sec - exiting 11:29:34 (4240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:29:35 (4240): No heartbeat from core client for 30 sec - exiting 11:29:36 (4240): No heartbeat from core client for 30 sec - exiting 11:29:37 (4240): No heartbeat from core client for 30 sec - exiting 11:29:38 (4240): No heartbeat from core client for 30 sec - exiting 11:29:39 (4240): No heartbeat from core client for 30 sec - exiting 11:29:40 (4240): No heartbeat from core client for 30 sec - exiting 11:29:41 (4240): No heartbeat from core client for 30 sec - exiting 11:29:44 (4240): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2508, iMonCtr=1 Model crash detected, will try to restart... 10:24:48 (852): No heartbeat from core client for 30 sec - exiting 10:24:50 (852): No heartbeat from core client for 30 sec - exiting 10:24:51 (852): No heartbeat from core client for 30 sec - exiting 10:24:52 (852): No heartbeat from core client for 30 sec - exiting 10:24:54 (852): No heartbeat from core client for 30 sec - exiting 10:24:55 (852): No heartbeat from core client for 30 sec - exiting 10:24:56 (852): No heartbeat from core client for 30 sec - exiting 10:24:57 (852): No heartbeat from core client for 30 sec - exiting 10:24:58 (852): No heartbeat from core client for 30 sec - exiting 10:24:59 (852): No heartbeat from core client for 30 sec - exiting 10:25:00 (852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:25:01 (852): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 10:38:16 (1644): No heartbeat from core client for 30 sec - exiting 10:38:17 (1644): No heartbeat from core client for 30 sec - exiting 10:38:22 (1644): No heartbeat from core client for 30 sec - exiting 10:38:23 (1644): No heartbeat from core client for 30 sec - exiting 10:38:24 (1644): No heartbeat from core client for 30 sec - exiting 10:38:25 (1644): No heartbeat from core client for 30 sec - exiting 10:38:26 (1644): No heartbeat from core client for 30 sec - exiting 10:38:27 (1644): No heartbeat from core client for 30 sec - exiting 10:38:28 (1644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:42:51 (2320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4548, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2560, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=328, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2172, iMonCtr=1 Model crash detected, will try to restart... 17:44:45 (2660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:05:24 (2072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:26 (2072): No heartbeat from core client for 30 sec - exiting 18:05:28 (2072): No heartbeat from core client for 30 sec - exiting 18:06:18 (4808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4236, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5032, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 08:05:31 (2448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:05:32 (2448): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2436, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1908, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 18:12:27 (3252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:12:28 (3252): No heartbeat from core client for 30 sec - exiting 18:12:29 (3252): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5148, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5496, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:31:58 (5316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:31:59 (5316): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2372, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2268, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3280, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4900, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:24:25 (2500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3856, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:18:22 (4084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:30:07 (4180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4672, iMonCtr=1 Model crash detected, will try to restart... 11:46:51 (2780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:01:15 (5860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... 09:43:38 (3796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:34:59 (4104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:35:00 (4104): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 13:54:46 (4508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:54:47 (4508): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:17:47 (916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:38:01 (3036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:38:02 (3036): No heartbeat from core client for 30 sec - exiting 14:38:03 (3036): No heartbeat from core client for 30 sec - exiting 14:38:04 (3036): No heartbeat from core client for 30 sec - exiting 14:38:05 (3036): No heartbeat from core client for 30 sec - exiting 14:38:06 (3036): No heartbeat from core client for 30 sec - exiting 14:38:08 (3036): No heartbeat from core client for 30 sec - exiting 14:38:09 (3036): No heartbeat from core client for 30 sec - exiting 14:38:10 (3036): No heartbeat from core client for 30 sec - exiting 14:38:11 (3036): No heartbeat from core client for 30 sec - exiting 14:38:12 (3036): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:47:30 (5348): No heartbeat from core client for 30 sec - exiting 11:47:33 (5348): No heartbeat from core client for 30 sec - exiting 11:47:34 (5348): No heartbeat from core client for 30 sec - exiting 11:47:35 (5348): No heartbeat from core client for 30 sec - exiting 11:47:36 (5348): No heartbeat from core client for 30 sec - exiting 11:47:38 (5348): No heartbeat from core client for 30 sec - exiting 11:47:39 (5348): No heartbeat from core client for 30 sec - exiting 11:47:40 (5348): No heartbeat from core client for 30 sec - exiting 11:47:42 (5348): No heartbeat from core client for 30 sec - exiting 11:47:43 (5348): No heartbeat from core client for 30 sec - exiting 11:47:44 (5348): No heartbeat from core client for 30 sec - exiting 11:47:45 (5348): No heartbeat from core client for 30 sec - exiting 11:47:46 (5348): No heartbeat from core client for 30 sec - exiting 11:47:47 (5348): No heartbeat from core client for 30 sec - exiting 11:47:48 (5348): No heartbeat from core client for 30 sec - exiting 11:47:50 (5348): No heartbeat from core client for 30 sec - exiting 11:47:51 (5348): No heartbeat from core client for 30 sec - exiting 11:47:52 (5348): No heartbeat from core client for 30 sec - exiting 11:47:53 (5348): No heartbeat from core client for 30 sec - exiting 11:47:54 (5348): No heartbeat from core client for 30 sec - exiting 11:47:55 (5348): No heartbeat from core client for 30 sec - exiting 11:47:56 (5348): No heartbeat from core client for 30 sec - exiting 11:47:57 (5348): No heartbeat from core client for 30 sec - exiting 11:47:58 (5348): No heartbeat from core client for 30 sec - exiting 11:48:00 (5348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:48:01 (5348): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:25:28 (3904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1112, iMonCtr=1 Model crash detected, will try to restart... 10:06:12 (5368): No heartbeat from core client for 30 sec - exiting 10:06:13 (5368): No heartbeat from core client for 30 sec - exiting 10:06:14 (5368): No heartbeat from core client for 30 sec - exiting 10:06:15 (5368): No heartbeat from core client for 30 sec - exiting 10:06:16 (5368): No heartbeat from core client for 30 sec - exiting 10:06:17 (5368): No heartbeat from core client for 30 sec - exiting 10:06:18 (5368): No heartbeat from core client for 30 sec - exiting 10:06:19 (5368): No heartbeat from core client for 30 sec - exiting 10:06:20 (5368): No heartbeat from core client for 30 sec - exiting 10:06:21 (5368): No heartbeat from core client for 30 sec - exiting 10:06:22 (5368): No heartbeat from core client for 30 sec - exiting 10:06:23 (5368): No heartbeat from core client for 30 sec - exiting 10:06:24 (5368): No heartbeat from core client for 30 sec - exiting 10:06:25 (5368): No heartbeat from core client for 30 sec - exiting 10:06:26 (5368): No heartbeat from core client for 30 sec - exiting 10:06:27 (5368): No heartbeat from core client for 30 sec - exiting 10:06:28 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:59:08 (2356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4308, iMonCtr=1 Model crash detected, will try to restart... 18:41:36 (5092): No heartbeat from core client for 30 sec - exiting 18:41:38 (5092): No heartbeat from core client for 30 sec - exiting 18:41:39 (5092): No heartbeat from core client for 30 sec - exiting 18:41:40 (5092): No heartbeat from core client for 30 sec - exiting 18:41:41 (5092): No heartbeat from core client for 30 sec - exiting 18:41:42 (5092): No heartbeat from core client for 30 sec - exiting 18:41:43 (5092): No heartbeat from core client for 30 sec - exiting 18:41:44 (5092): No heartbeat from core client for 30 sec - exiting 18:41:45 (5092): No heartbeat from core client for 30 sec - exiting 18:41:46 (5092): No heartbeat from core client for 30 sec - exiting 18:41:47 (5092): No heartbeat from core client for 30 sec - exiting 18:41:48 (5092): No heartbeat from core client for 30 sec - exiting 18:41:49 (5092): No heartbeat from core client for 30 sec - exiting 18:41:50 (5092): No heartbeat from core client for 30 sec - exiting 18:41:51 (5092): No heartbeat from core client for 30 sec - exiting 18:41:52 (5092): No heartbeat from core client for 30 sec - exiting 18:41:54 (5092): No heartbeat from core client for 30 sec - exiting 18:41:55 (5092): No heartbeat from core client for 30 sec - exiting 18:41:56 (5092): No heartbeat from core client for 30 sec - exiting 18:41:57 (5092): No heartbeat from core client for 30 sec - exiting 18:41:58 (5092): No heartbeat from core client for 30 sec - exiting 18:41:59 (5092): No heartbeat from core client for 30 sec - exiting 18:42:00 (5092): No heartbeat from core client for 30 sec - exiting 18:42:01 (5092): No heartbeat from core client for 30 sec - exiting 18:42:02 (5092): No heartbeat from core client for 30 sec - exiting 18:42:03 (5092): No heartbeat from core client for 30 sec - exiting 18:42:04 (5092): No heartbeat from core client for 30 sec - exiting 18:42:05 (5092): No heartbeat from core client for 30 sec - exiting 18:42:06 (5092): No heartbeat from core client for 30 sec - exiting 18:42:07 (5092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:42:08 (5092): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 19:59:17 (2788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2380, iMonCtr=1 Model crash detected, will try to restart... 18:40:43 (2236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1476, iMonCtr=1 Model crash detected, will try to restart... 20:00:41 (4644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:15:34 (1364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:05:21 (4688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:11:56 (1412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2704, iMonCtr=1 Model crash detected, will try to restart... 22:08:41 (2076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:45:35 (2768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:57:06 (2684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:41:07 (5968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:41:09 (5968): No heartbeat from core client for 30 sec - exiting 18:41:10 (5968): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 16:56:51 (5988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4836, iMonCtr=1 Model crash detected, will try to restart... 18:37:35 (1112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4252, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3728, iMonCtr=1 Model crash detected, will try to restart... 18:36:54 (936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:51:50 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:52:29 (4428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3532, iMonCtr=1 Model crash detected, will try to restart... 11:25:09 (2788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o721ko.pjj0c10 Error converting file to netcdf: dataout/o721ko.pij0c10 Error converting file to netcdf: dataout/o721ko.pfj0c10 Error converting file to netcdf: dataout/o721ko.pcj0c10 Error converting file to netcdf: dataout/o721ko.pbj0c10 Error converting file to netcdf: dataout/o721ko.paj0c10 Error converting file to netcdf: dataout/o721ka.phj0c10 Error converting file to netcdf: dataout/o721ka.pgj0c10 Error converting file to netcdf: dataout/o721ka.pej0c10 Error converting file to netcdf: dataout/o721ka.pdj0c10 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77AC1BD4 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... 17:38:40 (3140): No heartbeat from core client for 30 sec - exiting 17:38:42 (3140): No heartbeat from core client for 30 sec - exiting 17:38:43 (3140): No heartbeat from core client for 30 sec - exiting 17:38:44 (3140): No heartbeat from core client for 30 sec - exiting 17:38:46 (3140): No heartbeat from core client for 30 sec - exiting 17:38:47 (3140): No heartbeat from core client for 30 sec - exiting 17:38:48 (3140): No heartbeat from core client for 30 sec - exiting 17:38:49 (3140): No heartbeat from core client for 30 sec - exiting 17:38:50 (3140): No heartbeat from core client for 30 sec - exiting 17:38:51 (3140): No heartbeat from core client for 30 sec - exiting 17:38:52 (3140): No heartbeat from core client for 30 sec - exiting 17:38:53 (3140): No heartbeat from core client for 30 sec - exiting 17:38:55 (3140): No heartbeat from core client for 30 sec - exiting 17:38:56 (3140): No heartbeat from core client for 30 sec - exiting 17:38:57 (3140): No heartbeat from core client for 30 sec - exiting 17:38:58 (3140): No heartbeat from core client for 30 sec - exiting 17:38:59 (3140): No heartbeat from core client for 30 sec - exiting 17:39:00 (3140): No heartbeat from core client for 30 sec - exiting 17:39:02 (3140): No heartbeat from core client for 30 sec - exiting 17:39:04 (3140): No heartbeat from core client for 30 sec - exiting 17:39:04 (3140): No heartbeat from core client for 30 sec - exiting 17:39:05 (3140): No heartbeat from core client for 30 sec - exiting 17:39:07 (3140): No heartbeat from core client for 30 sec - exiting 17:39:08 (3140): No heartbeat from core client for 30 sec - exiting 17:39:09 (3140): No heartbeat from core client for 30 sec - exiting 17:39:10 (3140): No heartbeat from core client for 30 sec - exiting 17:39:11 (3140): No heartbeat from core client for 30 sec - exiting 17:39:12 (3140): No heartbeat from core client for 30 sec - exiting 17:39:13 (3140): No heartbeat from core client for 30 sec - exiting 17:39:14 (3140): No heartbeat from core client for 30 sec - exiting 17:39:15 (3140): No heartbeat from core client for 30 sec - exiting 17:39:17 (3140): No heartbeat from core client for 30 sec - exiting 17:39:18 (3140): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77876E9F read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... 17:39:19 (3140): No heartbeat from core client for 30 sec - exiting 17:39:20 (3140): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77851CAF read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Oct 2013 14:48:29	1280857	15929518	hadcm3n_o721_1980_40_008407763_0	259,200	856,484	3.3043
06 Oct 2013 15:25:52	1280857	15929518	hadcm3n_o721_1980_40_008407763_0	233,280	771,505	3.3072
23 Sep 2013 12:23:36	1280857	15929518	hadcm3n_o721_1980_40_008407763_0	207,360	668,864	3.2256
19 Sep 2013 14:23:02	1280857	15929518	hadcm3n_o721_1980_40_008407763_0	181,440	571,620	3.1505
12 Sep 2013 15:36:49	1280857	15929518	hadcm3n_o721_1980_40_008407763_0	155,520	481,448	3.0957
08 Sep 2013 13:33:25	1280857	15929518	hadcm3n_o721_1980_40_008407763_0	129,600	396,952	3.0629
02 Sep 2013 16:48:25	1280857	15929518	hadcm3n_o721_1980_40_008407763_0	103,680	316,027	3.0481
30 Aug 2013 16:29:44	1280857	15929518	hadcm3n_o721_1980_40_008407763_0	77,760	237,462	3.0538
27 Aug 2013 10:31:44	1280857	15929518	hadcm3n_o721_1980_40_008407763_0	51,840	152,982	2.9510
23 Aug 2013 16:38:14	1280857	15929518	hadcm3n_o721_1980_40_008407763_0	25,920	78,869	3.0428