Task 13350132

Name	hadcm3n_t5wu_1940_40_007443906_0
Workunit	7641409
Created	9 Sep 2011, 1:02:57 UTC
Sent	9 Sep 2011, 8:49:57 UTC
Report deadline	9 Dec 2011, 16:17:08 UTC
Received	22 Nov 2011, 17:13:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1045266
Run time	24 days 5 hours 29 min 50 sec
CPU time	23 days 11 hours 47 min 13 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.67 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4264, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4264, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4260, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4260, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4260, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4260, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4392, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4164, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4212, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4136, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2996, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4148, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4140, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4140, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4312, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4204, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2960, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2960, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7560, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4160, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4224, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3048, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3048, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3048, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3048, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4116, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1220, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4828, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4828, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4028, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4028, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4028, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7596, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2868, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4624, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3740, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1196, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2900, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2900, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2900, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3448, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on t5wuko.dag0c20 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3320, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4164, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3456, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3456, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5800, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2984, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1232, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4104, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2952, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2952, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2952, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2952, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3524, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4040, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3324, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4156, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6748, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4124, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7764, iMonCtr=1 Model crash detected, will try to restart... 18:26:33 (3740): No heartbeat from core client for 30 sec - exiting 18:26:34 (3740): No heartbeat from core client for 30 sec - exiting 18:26:35 (3740): No heartbeat from core client for 30 sec - exiting 18:26:36 (3740): No heartbeat from core client for 30 sec - exiting 18:26:38 (3740): No heartbeat from core client for 30 sec - exiting 18:26:39 (3740): No heartbeat from core client for 30 sec - exiting 18:26:40 (3740): No heartbeat from core client for 30 sec - exiting 18:26:41 (3740): No heartbeat from core client for 30 sec - exiting 18:26:42 (3740): No heartbeat from core client for 30 sec - exiting 18:26:43 (3740): No heartbeat from core client for 30 sec - exiting 18:26:44 (3740): No heartbeat from core client for 30 sec - exiting 18:26:45 (3740): No heartbeat from core client for 30 sec - exiting 18:26:46 (3740): No heartbeat from core client for 30 sec - exiting 18:26:47 (3740): No heartbeat from core client for 30 sec - exiting 18:26:48 (3740): No heartbeat from core client for 30 sec - exiting 18:26:50 (3740): No heartbeat from core client for 30 sec - exiting 18:26:51 (3740): No heartbeat from core client for 30 sec - exiting 18:26:52 (3740): No heartbeat from core client for 30 sec - exiting 18:26:53 (3740): No heartbeat from core client for 30 sec - exiting 18:26:54 (3740): No heartbeat from core client for 30 sec - exiting 18:26:55 (3740): No heartbeat from core client for 30 sec - exiting 18:26:56 (3740): No heartbeat from core client for 30 sec - exiting 18:26:57 (3740): No heartbeat from core client for 30 sec - exiting 18:26:58 (3740): No heartbeat from core client for 30 sec - exiting 18:26:59 (3740): No heartbeat from core client for 30 sec - exiting 18:27:00 (3740): No heartbeat from core client for 30 sec - exiting 18:27:02 (3740): No heartbeat from core client for 30 sec - exiting 18:27:03 (3740): No heartbeat from core client for 30 sec - exiting 18:27:04 (3740): No heartbeat from core client for 30 sec - exiting 18:27:05 (3740): No heartbeat from core client for 30 sec - exiting 18:27:06 (3740): No heartbeat from core client for 30 sec - exiting 18:27:07 (3740): No heartbeat from core client for 30 sec - exiting 18:27:08 (3740): No heartbeat from core client for 30 sec - exiting 18:27:09 (3740): No heartbeat from core client for 30 sec - exiting 18:27:10 (3740): No heartbeat from core client for 30 sec - exiting 18:27:11 (3740): No heartbeat from core client for 30 sec - exiting 18:27:12 (3740): No heartbeat from core client for 30 sec - exiting 18:27:14 (3740): No heartbeat from core client for 30 sec - exiting 18:27:15 (3740): No heartbeat from core client for 30 sec - exiting 18:27:16 (3740): No heartbeat from core client for 30 sec - exiting 18:27:17 (3740): No heartbeat from core client for 30 sec - exiting 18:27:18 (3740): No heartbeat from core client for 30 sec - exiting 18:27:19 (3740): No heartbeat from core client for 30 sec - exiting 18:27:20 (3740): No heartbeat from core client for 30 sec - exiting 18:27:21 (3740): No heartbeat from core client for 30 sec - exiting 18:27:22 (3740): No heartbeat from core client for 30 sec - exiting 18:27:23 (3740): No heartbeat from core client for 30 sec - exiting 18:27:24 (3740): No heartbeat from core client for 30 sec - exiting 18:27:26 (3740): No heartbeat from core client for 30 sec - exiting 18:27:27 (3740): No heartbeat from core client for 30 sec - exiting 18:27:28 (3740): No heartbeat from core client for 30 sec - exiting 18:27:29 (3740): No heartbeat from core client for 30 sec - exiting 18:27:30 (3740): No heartbeat from core client for 30 sec - exiting 18:27:31 (3740): No heartbeat from core client for 30 sec - exiting 18:27:32 (3740): No heartbeat from core client for 30 sec - exiting 18:27:33 (3740): No heartbeat from core client for 30 sec - exiting 18:27:34 (3740): No heartbeat from core client for 30 sec - exiting 18:27:35 (3740): No heartbeat from core client for 30 sec - exiting 18:27:36 (3740): No heartbeat from core client for 30 sec - exiting 18:27:38 (3740): No heartbeat from core client for 30 sec - exiting 18:27:39 (3740): No heartbeat from core client for 30 sec - exiting 18:27:40 (3740): No heartbeat from core client for 30 sec - exiting 18:27:41 (3740): No heartbeat from core client for 30 sec - exiting 18:27:42 (3740): No heartbeat from core client for 30 sec - exiting 18:27:43 (3740): No heartbeat from core client for 30 sec - exiting 18:27:44 (3740): No heartbeat from core client for 30 sec - exiting 18:27:45 (3740): No heartbeat from core client for 30 sec - exiting 18:27:46 (3740): No heartbeat from core client for 30 sec - exiting 18:27:47 (3740): No heartbeat from core client for 30 sec - exiting 18:27:48 (3740): No heartbeat from core client for 30 sec - exiting 18:27:50 (3740): No heartbeat from core client for 30 sec - exiting 18:27:51 (3740): No heartbeat from core client for 30 sec - exiting 18:27:52 (3740): No heartbeat from core client for 30 sec - exiting 18:27:53 (3740): No heartbeat from core client for 30 sec - exiting 18:27:54 (3740): No heartbeat from core client for 30 sec - exiting 18:27:55 (3740): No heartbeat from core client for 30 sec - exiting 18:27:56 (3740): No heartbeat from core client for 30 sec - exiting 18:27:57 (3740): No heartbeat from core client for 30 sec - exiting 18:27:58 (3740): No heartbeat from core client for 30 sec - exiting 18:27:59 (3740): No heartbeat from core client for 30 sec - exiting 18:28:00 (3740): No heartbeat from core client for 30 sec - exiting 18:28:02 (3740): No heartbeat from core client for 30 sec - exiting 18:28:03 (3740): No heartbeat from core client for 30 sec - exiting 18:28:04 (3740): No heartbeat from core client for 30 sec - exiting 18:28:05 (3740): No heartbeat from core client for 30 sec - exiting 18:28:06 (3740): No heartbeat from core client for 30 sec - exiting 18:28:07 (3740): No heartbeat from core client for 30 sec - exiting 18:28:08 (3740): No heartbeat from core client for 30 sec - exiting 18:28:09 (3740): No heartbeat from core client for 30 sec - exiting 18:28:10 (3740): No heartbeat from core client for 30 sec - exiting 18:28:11 (3740): No heartbeat from core client for 30 sec - exiting 18:28:13 (3740): No heartbeat from core client for 30 sec - exiting 18:28:14 (3740): No heartbeat from core client for 30 sec - exiting 18:28:15 (3740): No heartbeat from core client for 30 sec - exiting 18:28:16 (3740): No heartbeat from core client for 30 sec - exiting 18:28:17 (3740): No heartbeat from core client for 30 sec - exiting 18:28:18 (3740): No heartbeat from core client for 30 sec - exiting 18:28:19 (3740): No heartbeat from core client for 30 sec - exiting 18:28:20 (3740): No heartbeat from core client for 30 sec - exiting 18:28:21 (3740): No heartbeat from core client for 30 sec - exiting 18:28:22 (3740): No heartbeat from core client for 30 sec - exiting 18:28:23 (3740): No heartbeat from core client for 30 sec - exiting 18:28:25 (3740): No heartbeat from core client for 30 sec - exiting 18:28:26 (3740): No heartbeat from core client for 30 sec - exiting 18:28:27 (3740): No heartbeat from core client for 30 sec - exiting 18:28:28 (3740): No heartbeat from core client for 30 sec - exiting 18:28:29 (3740): No heartbeat from core client for 30 sec - exiting 18:28:30 (3740): No heartbeat from core client for 30 sec - exiting 18:28:31 (3740): No heartbeat from core client for 30 sec - exiting 18:28:32 (3740): No heartbeat from core client for 30 sec - exiting 18:28:33 (3740): No heartbeat from core client for 30 sec - exiting 18:28:34 (3740): No heartbeat from core client for 30 sec - exiting 18:28:35 (3740): No heartbeat from core client for 30 sec - exiting 18:28:37 (3740): No heartbeat from core client for 30 sec - exiting 18:28:38 (3740): No heartbeat from core client for 30 sec - exiting 18:28:39 (3740): No heartbeat from core client for 30 sec - exiting 18:28:40 (3740): No heartbeat from core client for 30 sec - exiting 18:28:41 (3740): No heartbeat from core client for 30 sec - exiting 18:28:42 (3740): No heartbeat from core client for 30 sec - exiting 18:28:43 (3740): No heartbeat from core client for 30 sec - exiting 18:28:44 (3740): No heartbeat from core client for 30 sec - exiting 18:28:45 (3740): No heartbeat from core client for 30 sec - exiting 18:28:46 (3740): No heartbeat from core client for 30 sec - exiting 18:28:47 (3740): No heartbeat from core client for 30 sec - exiting 18:28:49 (3740): No heartbeat from core client for 30 sec - exiting 18:28:50 (3740): No heartbeat from core client for 30 sec - exiting 18:28:51 (3740): No heartbeat from core client for 30 sec - exiting 18:28:52 (3740): No heartbeat from core client for 30 sec - exiting 18:28:53 (3740): No heartbeat from core client for 30 sec - exiting 18:28:54 (3740): No heartbeat from core client for 30 sec - exiting 18:28:55 (3740): No heartbeat from core client for 30 sec - exiting 18:28:56 (3740): No heartbeat from core client for 30 sec - exiting 18:28:57 (3740): No heartbeat from core client for 30 sec - exiting 18:28:58 (3740): No heartbeat from core client for 30 sec - exiting 18:28:59 (3740): No heartbeat from core client for 30 sec - exiting 18:29:01 (3740): No heartbeat from core client for 30 sec - exiting 18:29:02 (3740): No heartbeat from core client for 30 sec - exiting 18:29:03 (3740): No heartbeat from core client for 30 sec - exiting 18:29:04 (3740): No heartbeat from core client for 30 sec - exiting 18:29:05 (3740): No heartbeat from core client for 30 sec - exiting 18:29:06 (3740): No heartbeat from core client for 30 sec - exiting 18:29:07 (3740): No heartbeat from core client for 30 sec - exiting 18:29:08 (3740): No heartbeat from core client for 30 sec - exiting 18:29:09 (3740): No heartbeat from core client for 30 sec - exiting 18:29:10 (3740): No heartbeat from core client for 30 sec - exiting 18:29:11 (3740): No heartbeat from core client for 30 sec - exiting 18:29:13 (3740): No heartbeat from core client for 30 sec - exiting 18:29:14 (3740): No heartbeat from core client for 30 sec - exiting 18:29:15 (3740): No heartbeat from core client for 30 sec - exiting 18:29:16 (3740): No heartbeat from core client for 30 sec - exiting 18:29:17 (3740): No heartbeat from core client for 30 sec - exiting 18:29:18 (3740): No heartbeat from core client for 30 sec - exiting 18:29:19 (3740): No heartbeat from core client for 30 sec - exiting 18:29:20 (3740): No heartbeat from core client for 30 sec - exiting 18:29:21 (3740): No heartbeat from core client for 30 sec - exiting 18:29:22 (3740): No heartbeat from core client for 30 sec - exiting 18:29:23 (3740): No heartbeat from core client for 30 sec - exiting 18:29:25 (3740): No heartbeat from core client for 30 sec - exiting 18:29:26 (3740): No heartbeat from core client for 30 sec - exiting 18:29:27 (3740): No heartbeat from core client for 30 sec - exiting 18:29:28 (3740): No heartbeat from core client for 30 sec - exiting 18:29:29 (3740): No heartbeat from core client for 30 sec - exiting 18:29:30 (3740): No heartbeat from core client for 30 sec - exiting 18:29:31 (3740): No heartbeat from core client for 30 sec - exiting 18:29:32 (3740): No heartbeat from core client for 30 sec - exiting 18:29:33 (3740): No heartbeat from core client for 30 sec - exiting 18:29:34 (3740): No heartbeat from core client for 30 sec - exiting 18:29:35 (3740): No heartbeat from core client for 30 sec - exiting 18:29:37 (3740): No heartbeat from core client for 30 sec - exiting 18:29:38 (3740): No heartbeat from core client for 30 sec - exiting 18:29:39 (3740): No heartbeat from core client for 30 sec - exiting 18:29:40 (3740): No heartbeat from core client for 30 sec - exiting 18:29:41 (3740): No heartbeat from core client for 30 sec - exiting 18:29:42 (3740): No heartbeat from core client for 30 sec - exiting 18:29:43 (3740): No heartbeat from core client for 30 sec - exiting 18:29:44 (3740): No heartbeat from core client for 30 sec - exiting 18:29:45 (3740): No heartbeat from core client for 30 sec - exiting 18:29:46 (3740): No heartbeat from core client for 30 sec - exiting 18:29:48 (3740): No heartbeat from core client for 30 sec - exiting 18:29:49 (3740): No heartbeat from core client for 30 sec - exiting 18:29:50 (3740): No heartbeat from core client for 30 sec - exiting 18:29:51 (3740): No heartbeat from core client for 30 sec - exiting 18:29:52 (3740): No heartbeat from core client for 30 sec - exiting 18:29:53 (3740): No heartbeat from core client for 30 sec - exiting 18:29:54 (3740): No heartbeat from core client for 30 sec - exiting 18:29:55 (3740): No heartbeat from core client for 30 sec - exiting 18:29:56 (3740): No heartbeat from core client for 30 sec - exiting 18:29:57 (3740): No heartbeat from core client for 30 sec - exiting 18:29:58 (3740): No heartbeat from core client for 30 sec - exiting 18:30:00 (3740): No heartbeat from core client for 30 sec - exiting 18:30:01 (3740): No heartbeat from core client for 30 sec - exiting 18:30:02 (3740): No heartbeat from core client for 30 sec - exiting 18:30:03 (3740): No heartbeat from core client for 30 sec - exiting 18:30:04 (3740): No heartbeat from core client for 30 sec - exiting 18:30:05 (3740): No heartbeat from core client for 30 sec - exiting 18:30:06 (3740): No heartbeat from core client for 30 sec - exiting 18:30:07 (3740): No heartbeat from core client for 30 sec - exiting 18:30:08 (3740): No heartbeat from core client for 30 sec - exiting 18:30:09 (3740): No heartbeat from core client for 30 sec - exiting 18:30:10 (3740): No heartbeat from core client for 30 sec - exiting 18:30:12 (3740): No heartbeat from core client for 30 sec - exiting 18:30:13 (3740): No heartbeat from core client for 30 sec - exiting 18:30:14 (3740): No heartbeat from core client for 30 sec - exiting 18:30:15 (3740): No heartbeat from core client for 30 sec - exiting 18:30:16 (3740): No heartbeat from core client for 30 sec - exiting 18:30:17 (3740): No heartbeat from core client for 30 sec - exiting 18:30:18 (3740): No heartbeat from core client for 30 sec - exiting 18:30:19 (3740): No heartbeat from core client for 30 sec - exiting 18:30:20 (3740): No heartbeat from core client for 30 sec - exiting 18:30:21 (3740): No heartbeat from core client for 30 sec - exiting 18:30:22 (3740): No heartbeat from core client for 30 sec - exiting 18:30:24 (3740): No heartbeat from core client for 30 sec - exiting 18:30:25 (3740): No heartbeat from core client for 30 sec - exiting 18:30:26 (3740): No heartbeat from core client for 30 sec - exiting 18:30:27 (3740): No heartbeat from core client for 30 sec - exiting 18:30:28 (3740): No heartbeat from core client for 30 sec - exiting 18:30:29 (3740): No heartbeat from core client for 30 sec - exiting 18:30:30 (3740): No heartbeat from core client for 30 sec - exiting 18:30:31 (3740): No heartbeat from core client for 30 sec - exiting 18:30:32 (3740): No heartbeat from core client for 30 sec - exiting 18:30:33 (3740): No heartbeat from core client for 30 sec - exiting 18:30:34 (3740): No heartbeat from core client for 30 sec - exiting 18:30:36 (3740): No heartbeat from core client for 30 sec - exiting 18:30:37 (3740): No heartbeat from core client for 30 sec - exiting 18:30:38 (3740): No heartbeat from core client for 30 sec - exiting 18:30:39 (3740): No heartbeat from core client for 30 sec - exiting 18:30:40 (3740): No heartbeat from core client for 30 sec - exiting 18:30:41 (3740): No heartbeat from core client for 30 sec - exiting 18:30:42 (3740): No heartbeat from core client for 30 sec - exiting 18:30:43 (3740): No heartbeat from core client for 30 sec - exiting 18:30:44 (3740): No heartbeat from core client for 30 sec - exiting 18:30:45 (3740): No heartbeat from core client for 30 sec - exiting 18:30:46 (3740): No heartbeat from core client for 30 sec - exiting 18:30:48 (3740): No heartbeat from core client for 30 sec - exiting 18:30:49 (3740): No heartbeat from core client for 30 sec - exiting 18:30:50 (3740): No heartbeat from core client for 30 sec - exiting 18:30:51 (3740): No heartbeat from core client for 30 sec - exiting 18:30:52 (3740): No heartbeat from core client for 30 sec - exiting 18:30:53 (3740): No heartbeat from core client for 30 sec - exiting 18:30:54 (3740): No heartbeat from core client for 30 sec - exiting 18:30:55 (3740): No heartbeat from core client for 30 sec - exiting 18:30:56 (3740): No heartbeat from core client for 30 sec - exiting 18:30:57 (3740): No heartbeat from core client for 30 sec - exiting 18:30:58 (3740): No heartbeat from core client for 30 sec - exiting 18:31:00 (3740): No heartbeat from core client for 30 sec - exiting 18:31:01 (3740): No heartbeat from core client for 30 sec - exiting 18:31:02 (3740): No heartbeat from core client for 30 sec - exiting 18:31:03 (3740): No heartbeat from core client for 30 sec - exiting 18:31:04 (3740): No heartbeat from core client for 30 sec - exiting 18:31:05 (3740): No heartbeat from core client for 30 sec - exiting 18:31:06 (3740): No heartbeat from core client for 30 sec - exiting 18:31:07 (3740): No heartbeat from core client for 30 sec - exiting 18:31:08 (3740): No heartbeat from core client for 30 sec - exiting 18:31:09 (3740): No heartbeat from core client for 30 sec - exiting 18:31:11 (3740): No heartbeat from core client for 30 sec - exiting 18:31:12 (3740): No heartbeat from core client for 30 sec - exiting 18:31:13 (3740): No heartbeat from core client for 30 sec - exiting 18:31:14 (3740): No heartbeat from core client for 30 sec - exiting 18:31:15 (3740): No heartbeat from core client for 30 sec - exiting 18:31:16 (3740): No heartbeat from core client for 30 sec - exiting 18:31:17 (3740): No heartbeat from core client for 30 sec - exiting 18:31:18 (3740): No heartbeat from core client for 30 sec - exiting 18:31:19 (3740): No heartbeat from core client for 30 sec - exiting 18:31:20 (3740): No heartbeat from core client for 30 sec - exiting 18:31:21 (3740): No heartbeat from core client for 30 sec - exiting 18:31:23 (3740): No heartbeat from core client for 30 sec - exiting 18:31:24 (3740): No heartbeat from core client for 30 sec - exiting 18:31:25 (3740): No heartbeat from core client for 30 sec - exiting 18:31:26 (3740): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4268, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5792, selfPID=5792, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t5wuko.pjh3c10 Error converting file to netcdf: dataout/t5wuko.pih3c10 Error converting file to netcdf: dataout/t5wuko.pfh3c10 Error converting file to netcdf: dataout/t5wuka.phh3c10 Error converting file to netcdf: dataout/t5wuka.pgh3c10 Error converting file to netcdf: dataout/t5wuka.peh3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7752, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4116, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4116, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3884, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3884, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3132, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3032, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3032, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x775A6436 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Nov 2011 17:16:04	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	1,036,800	2,029,627	1.9576
21 Nov 2011 18:16:10	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	1,010,880	1,977,670	1.9564
20 Nov 2011 09:48:02	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	984,960	1,927,744	1.9572
19 Nov 2011 19:18:48	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	959,040	1,877,403	1.9576
18 Nov 2011 15:00:54	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	933,120	1,826,212	1.9571
15 Nov 2011 21:54:31	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	907,200	1,772,739	1.9541
15 Nov 2011 18:07:12	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	881,280	1,720,378	1.9521
15 Nov 2011 18:07:12	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	855,360	1,669,042	1.9513
15 Nov 2011 18:07:11	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	829,440	1,617,345	1.9499
08 Nov 2011 09:43:16	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	803,520	1,566,063	1.9490
06 Nov 2011 11:27:00	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	777,600	1,514,812	1.9481
05 Nov 2011 20:51:18	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	751,680	1,463,354	1.9468
03 Nov 2011 19:15:26	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	725,760	1,411,744	1.9452
01 Nov 2011 20:51:22	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	699,840	1,360,274	1.9437
31 Oct 2011 19:43:18	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	673,920	1,307,890	1.9407
31 Oct 2011 19:08:24	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	648,000	1,256,226	1.9386
31 Oct 2011 17:16:26	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	622,080	1,204,490	1.9362
31 Oct 2011 14:05:40	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	596,160	1,153,392	1.9347
31 Oct 2011 14:05:40	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	570,240	1,102,585	1.9335
31 Oct 2011 14:05:39	1045266	13350132	hadcm3n_t5wu_1940_40_007443906_0	544,320	1,049,957	1.9289