Task 13348605

Name	hadcm3n_t2ub_1940_40_007443310_1
Workunit	7640813
Created	8 Sep 2011, 23:34:50 UTC
Sent	19 Sep 2011, 16:14:03 UTC
Report deadline	19 Dec 2011, 23:41:14 UTC
Received	22 Nov 2011, 16:12:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1045266
Run time	18 days 15 hours 37 min 30 sec
CPU time	18 days 0 hours 33 min 41 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.67 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6596, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t2ubko.pje4c10 Error converting file to netcdf: dataout/t2ubko.pie4c10 Error converting file to netcdf: dataout/t2ubko.pfe4c10 Error converting file to netcdf: dataout/t2ubka.phe4c10 Error converting file to netcdf: dataout/t2ubka.pge4c10 Error converting file to netcdf: dataout/t2ubka.pee4c10 Error converting file to netcdf: dataout/t2ubka.pde4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3388, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3656, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4112, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2912, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2912, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2912, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3712, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1340, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3184, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4184, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3616, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3616, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3696, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3696, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3160, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1248, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3020, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3020, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3020, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3020, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4072, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1644, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4056, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1740, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4176, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4100, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:26:33 (2524): No heartbeat from core client for 30 sec - exiting 18:26:34 (2524): No heartbeat from core client for 30 sec - exiting 18:26:35 (2524): No heartbeat from core client for 30 sec - exiting 18:26:36 (2524): No heartbeat from core client for 30 sec - exiting 18:26:38 (2524): No heartbeat from core client for 30 sec - exiting 18:26:39 (2524): No heartbeat from core client for 30 sec - exiting 18:26:40 (2524): No heartbeat from core client for 30 sec - exiting 18:26:41 (2524): No heartbeat from core client for 30 sec - exiting 18:26:42 (2524): No heartbeat from core client for 30 sec - exiting 18:26:43 (2524): No heartbeat from core client for 30 sec - exiting 18:26:44 (2524): No heartbeat from core client for 30 sec - exiting 18:26:45 (2524): No heartbeat from core client for 30 sec - exiting 18:26:46 (2524): No heartbeat from core client for 30 sec - exiting 18:26:47 (2524): No heartbeat from core client for 30 sec - exiting 18:26:48 (2524): No heartbeat from core client for 30 sec - exiting 18:26:50 (2524): No heartbeat from core client for 30 sec - exiting 18:26:51 (2524): No heartbeat from core client for 30 sec - exiting 18:26:52 (2524): No heartbeat from core client for 30 sec - exiting 18:26:53 (2524): No heartbeat from core client for 30 sec - exiting 18:26:54 (2524): No heartbeat from core client for 30 sec - exiting 18:26:55 (2524): No heartbeat from core client for 30 sec - exiting 18:26:56 (2524): No heartbeat from core client for 30 sec - exiting 18:26:57 (2524): No heartbeat from core client for 30 sec - exiting 18:26:58 (2524): No heartbeat from core client for 30 sec - exiting 18:26:59 (2524): No heartbeat from core client for 30 sec - exiting 18:27:00 (2524): No heartbeat from core client for 30 sec - exiting 18:27:02 (2524): No heartbeat from core client for 30 sec - exiting 18:27:03 (2524): No heartbeat from core client for 30 sec - exiting 18:27:04 (2524): No heartbeat from core client for 30 sec - exiting 18:27:05 (2524): No heartbeat from core client for 30 sec - exiting 18:27:06 (2524): No heartbeat from core client for 30 sec - exiting 18:27:07 (2524): No heartbeat from core client for 30 sec - exiting 18:27:08 (2524): No heartbeat from core client for 30 sec - exiting 18:27:09 (2524): No heartbeat from core client for 30 sec - exiting 18:27:10 (2524): No heartbeat from core client for 30 sec - exiting 18:27:11 (2524): No heartbeat from core client for 30 sec - exiting 18:27:12 (2524): No heartbeat from core client for 30 sec - exiting 18:27:14 (2524): No heartbeat from core client for 30 sec - exiting 18:27:15 (2524): No heartbeat from core client for 30 sec - exiting 18:27:16 (2524): No heartbeat from core client for 30 sec - exiting 18:27:17 (2524): No heartbeat from core client for 30 sec - exiting 18:27:18 (2524): No heartbeat from core client for 30 sec - exiting 18:27:19 (2524): No heartbeat from core client for 30 sec - exiting 18:27:20 (2524): No heartbeat from core client for 30 sec - exiting 18:27:21 (2524): No heartbeat from core client for 30 sec - exiting 18:27:22 (2524): No heartbeat from core client for 30 sec - exiting 18:27:23 (2524): No heartbeat from core client for 30 sec - exiting 18:27:24 (2524): No heartbeat from core client for 30 sec - exiting 18:27:26 (2524): No heartbeat from core client for 30 sec - exiting 18:27:27 (2524): No heartbeat from core client for 30 sec - exiting 18:27:28 (2524): No heartbeat from core client for 30 sec - exiting 18:27:29 (2524): No heartbeat from core client for 30 sec - exiting 18:27:30 (2524): No heartbeat from core client for 30 sec - exiting 18:27:31 (2524): No heartbeat from core client for 30 sec - exiting 18:27:32 (2524): No heartbeat from core client for 30 sec - exiting 18:27:33 (2524): No heartbeat from core client for 30 sec - exiting 18:27:34 (2524): No heartbeat from core client for 30 sec - exiting 18:27:35 (2524): No heartbeat from core client for 30 sec - exiting 18:27:36 (2524): No heartbeat from core client for 30 sec - exiting 18:27:38 (2524): No heartbeat from core client for 30 sec - exiting 18:27:39 (2524): No heartbeat from core client for 30 sec - exiting 18:27:40 (2524): No heartbeat from core client for 30 sec - exiting 18:27:41 (2524): No heartbeat from core client for 30 sec - exiting 18:27:42 (2524): No heartbeat from core client for 30 sec - exiting 18:27:43 (2524): No heartbeat from core client for 30 sec - exiting 18:27:44 (2524): No heartbeat from core client for 30 sec - exiting 18:27:45 (2524): No heartbeat from core client for 30 sec - exiting 18:27:46 (2524): No heartbeat from core client for 30 sec - exiting 18:27:47 (2524): No heartbeat from core client for 30 sec - exiting 18:27:48 (2524): No heartbeat from core client for 30 sec - exiting 18:27:50 (2524): No heartbeat from core client for 30 sec - exiting 18:27:51 (2524): No heartbeat from core client for 30 sec - exiting 18:27:52 (2524): No heartbeat from core client for 30 sec - exiting 18:27:53 (2524): No heartbeat from core client for 30 sec - exiting 18:27:54 (2524): No heartbeat from core client for 30 sec - exiting 18:27:55 (2524): No heartbeat from core client for 30 sec - exiting 18:27:56 (2524): No heartbeat from core client for 30 sec - exiting 18:27:57 (2524): No heartbeat from core client for 30 sec - exiting 18:27:58 (2524): No heartbeat from core client for 30 sec - exiting 18:27:59 (2524): No heartbeat from core client for 30 sec - exiting 18:28:00 (2524): No heartbeat from core client for 30 sec - exiting 18:28:02 (2524): No heartbeat from core client for 30 sec - exiting 18:28:03 (2524): No heartbeat from core client for 30 sec - exiting 18:28:04 (2524): No heartbeat from core client for 30 sec - exiting 18:28:05 (2524): No heartbeat from core client for 30 sec - exiting 18:28:06 (2524): No heartbeat from core client for 30 sec - exiting 18:28:07 (2524): No heartbeat from core client for 30 sec - exiting 18:28:08 (2524): No heartbeat from core client for 30 sec - exiting 18:28:09 (2524): No heartbeat from core client for 30 sec - exiting 18:28:10 (2524): No heartbeat from core client for 30 sec - exiting 18:28:11 (2524): No heartbeat from core client for 30 sec - exiting 18:28:13 (2524): No heartbeat from core client for 30 sec - exiting 18:28:14 (2524): No heartbeat from core client for 30 sec - exiting 18:28:15 (2524): No heartbeat from core client for 30 sec - exiting 18:28:16 (2524): No heartbeat from core client for 30 sec - exiting 18:28:17 (2524): No heartbeat from core client for 30 sec - exiting 18:28:18 (2524): No heartbeat from core client for 30 sec - exiting 18:28:19 (2524): No heartbeat from core client for 30 sec - exiting 18:28:20 (2524): No heartbeat from core client for 30 sec - exiting 18:28:21 (2524): No heartbeat from core client for 30 sec - exiting 18:28:22 (2524): No heartbeat from core client for 30 sec - exiting 18:28:23 (2524): No heartbeat from core client for 30 sec - exiting 18:28:25 (2524): No heartbeat from core client for 30 sec - exiting 18:28:26 (2524): No heartbeat from core client for 30 sec - exiting 18:28:27 (2524): No heartbeat from core client for 30 sec - exiting 18:28:28 (2524): No heartbeat from core client for 30 sec - exiting 18:28:29 (2524): No heartbeat from core client for 30 sec - exiting 18:28:30 (2524): No heartbeat from core client for 30 sec - exiting 18:28:31 (2524): No heartbeat from core client for 30 sec - exiting 18:28:32 (2524): No heartbeat from core client for 30 sec - exiting 18:28:33 (2524): No heartbeat from core client for 30 sec - exiting 18:28:34 (2524): No heartbeat from core client for 30 sec - exiting 18:28:35 (2524): No heartbeat from core client for 30 sec - exiting 18:28:37 (2524): No heartbeat from core client for 30 sec - exiting 18:28:38 (2524): No heartbeat from core client for 30 sec - exiting 18:28:39 (2524): No heartbeat from core client for 30 sec - exiting 18:28:40 (2524): No heartbeat from core client for 30 sec - exiting 18:28:41 (2524): No heartbeat from core client for 30 sec - exiting 18:28:42 (2524): No heartbeat from core client for 30 sec - exiting 18:28:43 (2524): No heartbeat from core client for 30 sec - exiting 18:28:44 (2524): No heartbeat from core client for 30 sec - exiting 18:28:45 (2524): No heartbeat from core client for 30 sec - exiting 18:28:46 (2524): No heartbeat from core client for 30 sec - exiting 18:28:47 (2524): No heartbeat from core client for 30 sec - exiting 18:28:49 (2524): No heartbeat from core client for 30 sec - exiting 18:28:50 (2524): No heartbeat from core client for 30 sec - exiting 18:28:51 (2524): No heartbeat from core client for 30 sec - exiting 18:28:52 (2524): No heartbeat from core client for 30 sec - exiting 18:28:53 (2524): No heartbeat from core client for 30 sec - exiting 18:28:54 (2524): No heartbeat from core client for 30 sec - exiting 18:28:55 (2524): No heartbeat from core client for 30 sec - exiting 18:28:56 (2524): No heartbeat from core client for 30 sec - exiting 18:28:57 (2524): No heartbeat from core client for 30 sec - exiting 18:28:58 (2524): No heartbeat from core client for 30 sec - exiting 18:28:59 (2524): No heartbeat from core client for 30 sec - exiting 18:29:01 (2524): No heartbeat from core client for 30 sec - exiting 18:29:02 (2524): No heartbeat from core client for 30 sec - exiting 18:29:03 (2524): No heartbeat from core client for 30 sec - exiting 18:29:04 (2524): No heartbeat from core client for 30 sec - exiting 18:29:05 (2524): No heartbeat from core client for 30 sec - exiting 18:29:06 (2524): No heartbeat from core client for 30 sec - exiting 18:29:07 (2524): No heartbeat from core client for 30 sec - exiting 18:29:08 (2524): No heartbeat from core client for 30 sec - exiting 18:29:09 (2524): No heartbeat from core client for 30 sec - exiting 18:29:10 (2524): No heartbeat from core client for 30 sec - exiting 18:29:11 (2524): No heartbeat from core client for 30 sec - exiting 18:29:13 (2524): No heartbeat from core client for 30 sec - exiting 18:29:14 (2524): No heartbeat from core client for 30 sec - exiting 18:29:15 (2524): No heartbeat from core client for 30 sec - exiting 18:29:16 (2524): No heartbeat from core client for 30 sec - exiting 18:29:17 (2524): No heartbeat from core client for 30 sec - exiting 18:29:18 (2524): No heartbeat from core client for 30 sec - exiting 18:29:19 (2524): No heartbeat from core client for 30 sec - exiting 18:29:20 (2524): No heartbeat from core client for 30 sec - exiting 18:29:21 (2524): No heartbeat from core client for 30 sec - exiting 18:29:22 (2524): No heartbeat from core client for 30 sec - exiting 18:29:23 (2524): No heartbeat from core client for 30 sec - exiting 18:29:25 (2524): No heartbeat from core client for 30 sec - exiting 18:29:26 (2524): No heartbeat from core client for 30 sec - exiting 18:29:27 (2524): No heartbeat from core client for 30 sec - exiting 18:29:28 (2524): No heartbeat from core client for 30 sec - exiting 18:29:29 (2524): No heartbeat from core client for 30 sec - exiting 18:29:30 (2524): No heartbeat from core client for 30 sec - exiting 18:29:31 (2524): No heartbeat from core client for 30 sec - exiting 18:29:32 (2524): No heartbeat from core client for 30 sec - exiting 18:29:33 (2524): No heartbeat from core client for 30 sec - exiting 18:29:34 (2524): No heartbeat from core client for 30 sec - exiting 18:29:35 (2524): No heartbeat from core client for 30 sec - exiting 18:29:37 (2524): No heartbeat from core client for 30 sec - exiting 18:29:38 (2524): No heartbeat from core client for 30 sec - exiting 18:29:39 (2524): No heartbeat from core client for 30 sec - exiting 18:29:40 (2524): No heartbeat from core client for 30 sec - exiting 18:29:41 (2524): No heartbeat from core client for 30 sec - exiting 18:29:42 (2524): No heartbeat from core client for 30 sec - exiting 18:29:43 (2524): No heartbeat from core client for 30 sec - exiting 18:29:44 (2524): No heartbeat from core client for 30 sec - exiting 18:29:45 (2524): No heartbeat from core client for 30 sec - exiting 18:29:46 (2524): No heartbeat from core client for 30 sec - exiting 18:29:48 (2524): No heartbeat from core client for 30 sec - exiting 18:29:49 (2524): No heartbeat from core client for 30 sec - exiting 18:29:50 (2524): No heartbeat from core client for 30 sec - exiting 18:29:51 (2524): No heartbeat from core client for 30 sec - exiting 18:29:52 (2524): No heartbeat from core client for 30 sec - exiting 18:29:53 (2524): No heartbeat from core client for 30 sec - exiting 18:29:54 (2524): No heartbeat from core client for 30 sec - exiting 18:29:55 (2524): No heartbeat from core client for 30 sec - exiting 18:29:56 (2524): No heartbeat from core client for 30 sec - exiting 18:29:57 (2524): No heartbeat from core client for 30 sec - exiting 18:29:58 (2524): No heartbeat from core client for 30 sec - exiting 18:30:00 (2524): No heartbeat from core client for 30 sec - exiting 18:30:01 (2524): No heartbeat from core client for 30 sec - exiting 18:30:02 (2524): No heartbeat from core client for 30 sec - exiting 18:30:03 (2524): No heartbeat from core client for 30 sec - exiting 18:30:04 (2524): No heartbeat from core client for 30 sec - exiting 18:30:05 (2524): No heartbeat from core client for 30 sec - exiting 18:30:06 (2524): No heartbeat from core client for 30 sec - exiting 18:30:07 (2524): No heartbeat from core client for 30 sec - exiting 18:30:08 (2524): No heartbeat from core client for 30 sec - exiting 18:30:09 (2524): No heartbeat from core client for 30 sec - exiting 18:30:10 (2524): No heartbeat from core client for 30 sec - exiting 18:30:12 (2524): No heartbeat from core client for 30 sec - exiting 18:30:13 (2524): No heartbeat from core client for 30 sec - exiting 18:30:14 (2524): No heartbeat from core client for 30 sec - exiting 18:30:15 (2524): No heartbeat from core client for 30 sec - exiting 18:30:16 (2524): No heartbeat from core client for 30 sec - exiting 18:30:17 (2524): No heartbeat from core client for 30 sec - exiting 18:30:18 (2524): No heartbeat from core client for 30 sec - exiting 18:30:19 (2524): No heartbeat from core client for 30 sec - exiting 18:30:20 (2524): No heartbeat from core client for 30 sec - exiting 18:30:21 (2524): No heartbeat from core client for 30 sec - exiting 18:30:22 (2524): No heartbeat from core client for 30 sec - exiting 18:30:24 (2524): No heartbeat from core client for 30 sec - exiting 18:30:25 (2524): No heartbeat from core client for 30 sec - exiting 18:30:26 (2524): No heartbeat from core client for 30 sec - exiting 18:30:27 (2524): No heartbeat from core client for 30 sec - exiting 18:30:28 (2524): No heartbeat from core client for 30 sec - exiting 18:30:29 (2524): No heartbeat from core client for 30 sec - exiting 18:30:30 (2524): No heartbeat from core client for 30 sec - exiting 18:30:31 (2524): No heartbeat from core client for 30 sec - exiting 18:30:32 (2524): No heartbeat from core client for 30 sec - exiting 18:30:33 (2524): No heartbeat from core client for 30 sec - exiting 18:30:34 (2524): No heartbeat from core client for 30 sec - exiting 18:30:36 (2524): No heartbeat from core client for 30 sec - exiting 18:30:37 (2524): No heartbeat from core client for 30 sec - exiting 18:30:38 (2524): No heartbeat from core client for 30 sec - exiting 18:30:39 (2524): No heartbeat from core client for 30 sec - exiting 18:30:40 (2524): No heartbeat from core client for 30 sec - exiting 18:30:41 (2524): No heartbeat from core client for 30 sec - exiting 18:30:42 (2524): No heartbeat from core client for 30 sec - exiting 18:30:43 (2524): No heartbeat from core client for 30 sec - exiting 18:30:44 (2524): No heartbeat from core client for 30 sec - exiting 18:30:45 (2524): No heartbeat from core client for 30 sec - exiting 18:30:46 (2524): No heartbeat from core client for 30 sec - exiting 18:30:48 (2524): No heartbeat from core client for 30 sec - exiting 18:30:49 (2524): No heartbeat from core client for 30 sec - exiting 18:30:50 (2524): No heartbeat from core client for 30 sec - exiting 18:30:51 (2524): No heartbeat from core client for 30 sec - exiting 18:30:52 (2524): No heartbeat from core client for 30 sec - exiting 18:30:53 (2524): No heartbeat from core client for 30 sec - exiting 18:30:54 (2524): No heartbeat from core client for 30 sec - exiting 18:30:55 (2524): No heartbeat from core client for 30 sec - exiting 18:30:56 (2524): No heartbeat from core client for 30 sec - exiting 18:30:57 (2524): No heartbeat from core client for 30 sec - exiting 18:30:58 (2524): No heartbeat from core client for 30 sec - exiting 18:31:00 (2524): No heartbeat from core client for 30 sec - exiting 18:31:01 (2524): No heartbeat from core client for 30 sec - exiting 18:31:02 (2524): No heartbeat from core client for 30 sec - exiting 18:31:03 (2524): No heartbeat from core client for 30 sec - exiting 18:31:04 (2524): No heartbeat from core client for 30 sec - exiting 18:31:05 (2524): No heartbeat from core client for 30 sec - exiting 18:31:06 (2524): No heartbeat from core client for 30 sec - exiting 18:31:07 (2524): No heartbeat from core client for 30 sec - exiting 18:31:08 (2524): No heartbeat from core client for 30 sec - exiting 18:31:09 (2524): No heartbeat from core client for 30 sec - exiting 18:31:11 (2524): No heartbeat from core client for 30 sec - exiting 18:31:12 (2524): No heartbeat from core client for 30 sec - exiting 18:31:13 (2524): No heartbeat from core client for 30 sec - exiting 18:31:14 (2524): No heartbeat from core client for 30 sec - exiting 18:31:15 (2524): No heartbeat from core client for 30 sec - exiting 18:31:16 (2524): No heartbeat from core client for 30 sec - exiting 18:31:17 (2524): No heartbeat from core client for 30 sec - exiting 18:31:18 (2524): No heartbeat from core client for 30 sec - exiting 18:31:19 (2524): No heartbeat from core client for 30 sec - exiting 18:31:20 (2524): No heartbeat from core client for 30 sec - exiting 18:31:21 (2524): No heartbeat from core client for 30 sec - exiting 18:31:23 (2524): No heartbeat from core client for 30 sec - exiting 18:31:24 (2524): No heartbeat from core client for 30 sec - exiting 18:31:25 (2524): No heartbeat from core client for 30 sec - exiting 18:31:26 (2524): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4020, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4140, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5620, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3928, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3928, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1224, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8032, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1200, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x775A6436 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Nov 2011 14:15:19	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	777,600	1,557,214	2.0026
20 Nov 2011 20:09:51	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	751,680	1,503,161	1.9997
20 Nov 2011 05:42:11	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	725,760	1,451,523	2.0000
19 Nov 2011 11:49:31	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	699,840	1,399,942	2.0004
18 Nov 2011 12:25:11	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	673,920	1,347,596	1.9996
15 Nov 2011 19:57:15	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	648,000	1,295,433	1.9991
15 Nov 2011 18:07:08	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	622,080	1,242,321	1.9970
15 Nov 2011 18:07:08	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	596,160	1,191,079	1.9979
09 Nov 2011 20:22:01	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	570,240	1,138,229	1.9961
07 Nov 2011 18:09:43	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	544,320	1,085,633	1.9945
06 Nov 2011 07:01:43	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	518,400	1,033,473	1.9936
05 Nov 2011 16:16:18	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	492,480	980,312	1.9906
02 Nov 2011 19:06:52	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	466,560	927,542	1.9880
31 Oct 2011 22:11:11	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	440,640	873,670	1.9827
31 Oct 2011 19:22:53	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	414,720	821,655	1.9812
31 Oct 2011 18:45:22	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	388,800	769,273	1.9786
31 Oct 2011 16:37:47	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	362,880	717,282	1.9766
31 Oct 2011 14:05:38	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	336,960	664,917	1.9733
31 Oct 2011 14:05:38	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	311,040	615,306	1.9782
31 Oct 2011 14:05:38	1045266	13348605	hadcm3n_t2ub_1940_40_007443310_1	285,120	564,006	1.9781