Task 13347395

Name	hadcm3n_t0z1_1940_40_007442812_1
Workunit	7640315
Created	8 Sep 2011, 22:17:59 UTC
Sent	8 Sep 2011, 22:19:38 UTC
Report deadline	9 Dec 2011, 5:46:49 UTC
Received	16 Jun 2012, 4:00:05 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1167540
Run time	5 days 7 hours 4 min 50 sec
CPU time	4 days 15 hours 25 min 46 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.63 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2168, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:32:26 (3848): No heartbeat from core client for 30 sec - exiting 07:32:27 (3848): No heartbeat from core client for 30 sec - exiting 07:32:28 (3848): No heartbeat from core client for 30 sec - exiting 07:32:29 (3848): No heartbeat from core client for 30 sec - exiting 07:32:30 (3848): No heartbeat from core client for 30 sec - exiting 07:32:31 (3848): No heartbeat from core client for 30 sec - exiting 07:32:32 (3848): No heartbeat from core client for 30 sec - exiting 07:32:33 (3848): No heartbeat from core client for 30 sec - exiting 07:32:34 (3848): No heartbeat from core client for 30 sec - exiting 07:32:35 (3848): No heartbeat from core client for 30 sec - exiting 07:32:36 (3848): No heartbeat from core client for 30 sec - exiting 07:32:37 (3848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:52:45 (3372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:19:27 (2764): No heartbeat from core client for 30 sec - exiting 05:19:28 (2764): No heartbeat from core client for 30 sec - exiting 05:19:29 (2764): No heartbeat from core client for 30 sec - exiting 05:19:30 (2764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:03:49 (5728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:34:40 (5460): No heartbeat from core client for 30 sec - exiting 23:34:41 (5460): No heartbeat from core client for 30 sec - exiting 23:34:42 (5460): No heartbeat from core client for 30 sec - exiting 23:34:43 (5460): No heartbeat from core client for 30 sec - exiting 23:34:44 (5460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CWorker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:34:41 (4904): No heartbeat from core client for 30 sec - exiting 09:34:42 (4904): No heartbeat from core client for 30 sec - exiting 09:34:43 (4904): No heartbeat from core client for 30 sec - exiting 09:34:44 (4904): No heartbeat from core client for 30 sec - exiting 09:34:45 (4904): No heartbeat from core client for 30 sec - exiting 09:34:46 (4904): No heartbeat from core client for 30 sec - exiting 09:34:47 (4904): No heartbeat from core client for 30 sec - exiting 09:34:48 (4904): No heartbeat from core client for 30 sec - exiting 09:34:49 (4904): No heartbeat from core client for 30 sec - exiting 09:34:50 (4904): No heartbeat from core client for 30 sec - exiting 09:34:51 (4904): No heartbeat from core client for 30 sec - exiting 09:34:52 (4904): No heartbeat from core client for 30 sec - exiting 09:34:53 (4904): No heartbeat from core client for 30 sec - exiting 09:34:54 (4904): No heartbeat from core client for 30 sec - exiting 09:34:55 (4904): No heartbeat from core client for 30 sec - exiting 09:34:56 (4904): No heartbeat from core client for 30 sec - exiting 09:34:57 (4904): No heartbeat from core client for 30 sec - exiting 09:34:58 (4904): No heartbeat from core client for 30 sec - exiting 09:34:59 (4904): No heartbeat from core client for 30 sec - exiting 09:35:00 (4904): No heartbeat from core client for 30 sec - exiting 09:35:01 (4904): No heartbeat from core client for 30 sec - exiting 09:35:02 (4904): No heartbeat from core client for 30 sec - exiting 09:35:03 (4904): No heartbeat from core client for 30 sec - exiting 09:35:04 (4904): No heartbeat from core client for 30 sec - exiting 09:35:05 (4904): No heartbeat from core client for 30 sec - exiting 09:35:06 (4904): No heartbeat from core client for 30 sec - exiting 09:35:07 (4904): No heartbeat from core client for 30 sec - exiting 09:35:08 (4904): No heartbeat from core client for 30 sec - exiting 09:35:09 (4904): No heartbeat from core client for 30 sec - exiting 09:35:10 (4904): No heartbeat from core client for 30 sec - exiting 09:35:11 (4904): No heartbeat from core client for 30 sec - exiting 09:35:12 (4904): No heartbeat from core client for 30 sec - exiting 09:35:13 (4904): No heartbeat from core client for 30 sec - exiting 09:35:14 (4904): No heartbeat from core client for 30 sec - exiting 09:35:15 (4904): No heartbeat from core client for 30 sec - exiting 09:35:16 (4904): No heartbeat from core client for 30 sec - exiting 09:35:17 (4904): No heartbeat from core client for 30 sec - exiting 09:35:18 (4904): No heartbeat from core client for 30 sec - exiting 09:35:19 (4904): No heartbeat from core client for 30 sec - exiting 09:35:20 (4904): No heartbeat from core client for 30 sec - exiting 09:35:21 (4904): No heartbeat from core client for 30 sec - exiting 09:35:22 (4904): No heartbeat from core client for 30 sec - exiting 09:35:23 (4904): No heartbeat from core client for 30 sec - exiting 09:35:24 (4904): No heartbeat from core client for 30 sec - exiting 09:35:25 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:35:26 (4904): No heartbeat from core client for 30 sec - exiting 09:35:27 (4904): No heartbeat from core client for 30 sec - exiting 09:35:28 (4904): No heartbeat from core client for 30 sec - exiting 09:35:29 (4904): No heartbeat from core client for 30 sec - exiting 09:35:30 (4904): No heartbeat from core client for 30 sec - exiting 09:35:31 (4904): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:33:06 (3076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CNo Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2728, selfPID=2728, iMonCtr=1 forrtl: Access is denied. CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4596, iMonCtr=1 Model crash detected, will try to restart... 18:13:49 (5624): No heartbeat from core client for 30 sec - exiting 18:13:50 (5624): No heartbeat from core client for 30 sec - exiting 18:13:51 (5624): No heartbeat from core client for 30 sec - exiting 18:13:52 (5624): No heartbeat from core client for 30 sec - exiting 18:13:53 (5624): No heartbeat from core client for 30 sec - exiting 18:13:54 (5624): No heartbeat from core client for 30 sec - exiting 18:13:55 (5624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3272, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3272, iMonCtr=1 Model crash detected, will try to restart... 03:31:07 (4828): No heartbeat from core client for 30 sec - exiting 03:31:08 (4828): No heartbeat from core client for 30 sec - exiting 03:31:09 (4828): No heartbeat from core client for 30 sec - exiting 03:31:10 (4828): No heartbeat from core client for 30 sec - exiting 03:31:11 (4828): No heartbeat from core client for 30 sec - exiting 03:31:12 (4828): No heartbeat from core client for 30 sec - exiting 03:31:13 (4828): No heartbeat from core client for 30 sec - exiting 03:31:14 (4828): No heartbeat from core client for 30 sec - exiting 03:31:15 (4828): No heartbeat from core client for 30 sec - exiting 03:31:16 (4828): No heartbeat from core client for 30 sec - exiting 03:31:17 (4828): No heartbeat from core client for 30 sec - exiting 03:31:18 (4828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:29:22 (3264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:57:02 (2916): No heartbeat from core client for 30 sec - exiting 12:57:03 (2916): No heartbeat from core client for 30 sec - exiting 12:57:04 (2916): No heartbeat from core client for 30 sec - exiting 12:57:05 (2916): No heartbeat from core client for 30 sec - exiting 12:57:06 (2916): No heartbeat from core client for 30 sec - exiting 12:57:07 (2916): No heartbeat from core client for 30 sec - exiting 12:57:08 (2916): No heartbeat from core client for 30 sec - exiting 12:57:09 (2916): No heartbeat from core client for 30 sec - exiting 12:57:10 (2916): No heartbeat from core client for 30 sec - exiting 12:57:11 (2916): No heartbeat from core client for 30 sec - exiting 12:57:12 (2916): No heartbeat from core client for 30 sec - exiting 12:57:13 (2916): No heartbeat from core client for 30 sec - exiting 12:57:14 (2916): No heartbeat from core client for 30 sec - exiting 12:57:15 (2916): No heartbeat from core client for 30 sec - exiting 12:57:16 (2916): No heartbeat from core client for 30 sec - exiting 12:57:17 (2916): No heartbeat from core client for 30 sec - exiting 12:57:18 (2916): No heartbeat from core client for 30 sec - exiting 12:57:19 (2916): No heartbeat from core client for 30 sec - exiting 12:57:20 (2916): No heartbeat from core client for 30 sec - exiting 12:57:21 (2916): No heartbeat from core client for 30 sec - exiting 12:57:22 (2916): No heartbeat from core client for 30 sec - exiting 12:57:23 (2916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:09:57 (5528): No heartbeat from core client for 30 sec - exiting 17:09:58 (5528): No heartbeat from core client for 30 sec - exiting 17:09:59 (5528): No heartbeat from core client for 30 sec - exiting 17:10:00 (5528): No heartbeat from core client for 30 sec - exiting 17:10:01 (5528): No heartbeat from core client for 30 sec - exiting 17:10:02 (5528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:10:03 (5528): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4480, iMonCtr=1 Model crash detected, will try to restart... 03:27:55 (2932): No heartbeat from core client for 30 sec - exiting 03:27:56 (2932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3484, iMonCtr=1 Model crash detected, will try to restart... 13:09:22 (6004): No heartbeat from core client for 30 sec - exiting 13:09:23 (6004): No heartbeat from core client for 30 sec - exiting 13:09:24 (6004): No heartbeat from core client for 30 sec - exiting 13:09:25 (6004): No heartbeat from core client for 30 sec - exiting 13:09:26 (6004): No heartbeat from core client for 30 sec - exiting 13:09:27 (6004): No heartbeat from core client for 30 sec - exiting 13:09:28 (6004): No heartbeat from core client for 30 sec - exiting 13:09:29 (6004): No heartbeat from core client for 30 sec - exiting 13:09:30 (6004): No heartbeat from core client for 30 sec - exiting 13:09:31 (6004): No heartbeat from core client for 30 sec - exiting 13:09:32 (6004): No heartbeat from core client for 30 sec - exiting 13:09:33 (6004): No heartbeat from core client for 30 sec - exiting 13:09:34 (6004): No heartbeat from core client for 30 sec - exiting 13:09:35 (6004): No heartbeat from core client for 30 sec - exiting 13:09:36 (6004): No heartbeat from core client for 30 sec - exiting 13:09:37 (6004): No heartbeat from core client for 30 sec - exiting 13:09:38 (6004): No heartbeat from core client for 30 sec - exiting 13:09:39 (6004): No heartbeat from core client for 30 sec - exiting 13:09:40 (6004): No heartbeat from core client for 30 sec - exiting 13:09:41 (6004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:08:52 (4716): No heartbeat from core client for 30 sec - exiting 14:08:53 (4716): No heartbeat from core client for 30 sec - exiting 14:08:54 (4716): No heartbeat from core client for 30 sec - exiting 14:08:55 (4716): No heartbeat from core client for 30 sec - exiting 14:08:56 (4716): No heartbeat from core client for 30 sec - exiting 14:08:57 (4716): No heartbeat from core client for 30 sec - exiting 14:08:58 (4716): No heartbeat from core client for 30 sec - exiting 14:08:59 (4716): No heartbeat from core client for 30 sec - exiting 14:09:00 (4716): No heartbeat from core client for 30 sec - exiting 14:09:01 (4716): No heartbeat from core client for 30 sec - exiting 14:09:02 (4716): No heartbeat from core client for 30 sec - exiting 14:09:03 (4716): No heartbeat from core client for 30 sec - exiting 14:09:04 (4716): No heartbeat from core client for 30 sec - exiting 14:09:05 (4716): No heartbeat from core client for 30 sec - exiting 14:09:06 (4716): No heartbeat from core client for 30 sec - exiting 14:09:07 (4716): No heartbeat from core client for 30 sec - exiting 14:09:08 (4716): No heartbeat from core client for 30 sec - exiting 14:09:09 (4716): No heartbeat from core client for 30 sec - exiting 14:09:10 (4716): No heartbeat from core client for 30 sec - exiting 14:09:11 (4716): No heartbeat from core client for 30 sec - exiting 14:09:12 (4716): No heartbeat from core client for 30 sec - exiting 14:09:13 (4716): No heartbeat from core client for 30 sec - exiting 14:09:14 (4716): No heartbeat from core client for 30 sec - exiting 14:09:15 (4716): No heartbeat from core client for 30 sec - exiting 14:09:16 (4716): No heartbeat from core client for 30 sec - exiting 14:09:17 (4716): No heartbeat from core client for 30 sec - exiting 14:09:18 (4716): No heartbeat from core client for 30 sec - exiting 14:09:19 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:09:20 (4716): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:56:22 (4196): No heartbeat from core client for 30 sec - exiting 15:56:23 (4196): No heartbeat from core client for 30 sec - exiting 15:56:24 (4196): No heartbeat from core client for 30 sec - exiting 15:56:25 (4196): No heartbeat from core client for 30 sec - exiting 15:56:26 (4196): No heartbeat from core client for 30 sec - exiting 15:56:27 (4196): No heartbeat from core client for 30 sec - exiting 15:56:28 (4196): No heartbeat from core client for 30 sec - exiting 15:56:29 (4196): No heartbeat from core client for 30 sec - exiting 15:56:30 (4196): No heartbeat from core client for 30 sec - exiting 15:56:31 (4196): No heartbeat from core client for 30 sec - exiting 15:56:32 (4196): No heartbeat from core client for 30 sec - exiting 15:56:33 (4196): No heartbeat from core client for 30 sec - exiting 15:56:34 (4196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:56:35 (4196): No heartbeat from core client for 30 sec - exiting 15:56:36 (4196): No heartbeat from core client for 30 sec - exiting 15:56:37 (4196): No heartbeat from core client for 30 sec - exiting 15:56:38 (4196): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4688, iMonCtr=1 Model crash detected, will try to restart... 18:25:48 (3196): No heartbeat from core client for 30 sec - exiting 18:25:49 (3196): No heartbeat from core client for 30 sec - exiting 18:25:50 (3196): No heartbeat from core client for 30 sec - exiting 18:25:51 (3196): No heartbeat from core client for 30 sec - exiting 18:25:52 (3196): No heartbeat from core client for 30 sec - exiting 18:25:53 (3196): No heartbeat from core client for 30 sec - exiting 18:25:54 (3196): No heartbeat from core client for 30 sec - exiting 18:25:56 (3196): No heartbeat from core client for 30 sec - exiting 18:25:57 (3196): No heartbeat from core client for 30 sec - exiting 18:25:58 (3196): No heartbeat from core client for 30 sec - exiting 18:25:59 (3196): No heartbeat from core client for 30 sec - exiting 18:26:00 (3196): No heartbeat from core client for 30 sec - exiting 18:26:01 (3196): No heartbeat from core client for 30 sec - exiting 18:26:02 (3196): No heartbeat from core client for 30 sec - exiting 18:26:03 (3196): No heartbeat from core client for 30 sec - exiting 18:26:04 (3196): No heartbeat from core client for 30 sec - exiting 18:26:05 (3196): No heartbeat from core client for 30 sec - exiting 18:26:06 (3196): No heartbeat from core client for 30 sec - exiting 18:26:08 (3196): No heartbeat from core client for 30 sec - exiting 18:26:09 (3196): No heartbeat from core client for 30 sec - exiting 18:26:10 (3196): No heartbeat from core client for 30 sec - exiting 18:26:11 (3196): No heartbeat from core client for 30 sec - exiting 18:26:12 (3196): No heartbeat from core client for 30 sec - exiting 18:26:13 (3196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:19:16 (4264): No heartbeat from core client for 30 sec - exiting 13:19:17 (4264): No heartbeat from core client for 30 sec - exiting 13:19:18 (4264): No heartbeat from core client for 30 sec - exiting 13:19:19 (4264): No heartbeat from core client for 30 sec - exiting 13:19:20 (4264): No heartbeat from core client for 30 sec - exiting 13:19:21 (4264): No heartbeat from core client for 30 sec - exiting 13:19:22 (4264): No heartbeat from core client for 30 sec - exiting 13:19:23 (4264): No heartbeat from core client for 30 sec - exiting 13:19:24 (4264): No heartbeat from core client for 30 sec - exiting 13:19:25 (4264): No heartbeat from core client for 30 sec - exiting 13:19:26 (4264): No heartbeat from core client for 30 sec - exiting 13:19:27 (4264): No heartbeat from core client for 30 sec - exiting 13:19:28 (4264): No heartbeat from core client for 30 sec - exiting 13:19:29 (4264): No heartbeat from core client for 30 sec - exiting 13:19:30 (4264): No heartbeat from core client for 30 sec - exiting 13:19:31 (4264): No heartbeat from core client for 30 sec - exiting 13:19:32 (4264): No heartbeat from core client for 30 sec - exiting 13:19:33 (4264): No heartbeat from core client for 30 sec - exiting 13:19:34 (4264): No heartbeat from core client for 30 sec - exiting 13:19:35 (4264): No heartbeat from core client for 30 sec - exiting 13:19:36 (4264): No heartbeat from core client for 30 sec - exiting 13:19:37 (4264): No heartbeat from core client for 30 sec - exiting 13:19:38 (4264): No heartbeat from core client for 30 sec - exiting 13:19:39 (4264): No heartbeat from core client for 30 sec - exiting 13:19:40 (4264): No heartbeat from core client for 30 sec - exiting 13:19:41 (4264): No heartbeat from core client for 30 sec - exiting 13:19:42 (4264): No heartbeat from core client for 30 sec - exiting 13:19:43 (4264): No heartbeat from core client for 30 sec - exiting 13:19:44 (4264): No heartbeat from core client for 30 sec - exiting 13:19:45 (4264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:19:46 (4264): No heartbeat from core client for 30 sec - exiting 13:19:47 (4264): No heartbeat from core client for 30 sec - exiting 13:19:48 (4264): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5628, iMonCtr=1 Model crash detected, will try to restart... 06:12:16 (4876): No heartbeat from core client for 30 sec - exiting 06:12:17 (4876): No heartbeat from core client for 30 sec - exiting 06:12:18 (4876): No heartbeat from core client for 30 sec - exiting 06:12:19 (4876): No heartbeat from core client for 30 sec - exiting 06:12:20 (4876): No heartbeat from core client for 30 sec - exiting 06:12:21 (4876): No heartbeat from core client for 30 sec - exiting 06:12:22 (4876): No heartbeat from core client for 30 sec - exiting 06:12:23 (4876): No heartbeat from core client for 30 sec - exiting 06:12:24 (4876): No heartbeat from core client for 30 sec - exiting 06:12:25 (4876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:46:05 (5772): No heartbeat from core client for 30 sec - exiting 10:46:06 (5772): No heartbeat from core client for 30 sec - exiting 10:46:07 (5772): No heartbeat from core client for 30 sec - exiting 10:46:08 (5772): No heartbeat from core client for 30 sec - exiting 10:46:09 (5772): No heartbeat from core client for 30 sec - exiting 10:46:10 (5772): No heartbeat from core client for 30 sec - exiting 10:46:11 (5772): No heartbeat from core client for 30 sec - exiting 10:46:12 (5772): No heartbeat from core client for 30 sec - exiting 10:46:13 (5772): No heartbeat from core client for 30 sec - exiting 10:46:14 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 11:56:17 (2960): No heartbeat from core client for 30 sec - exiting 11:56:18 (2960): No heartbeat from core client for 30 sec - exiting 11:56:19 (2960): No heartbeat from core client for 30 sec - exiting 11:56:20 (2960): No heartbeat from core client for 30 sec - exiting 11:56:21 (2960): No heartbeat from core client for 30 sec - exiting 11:56:22 (2960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:37:58 (5712): No heartbeat from core client for 30 sec - exiting 20:37:59 (5712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:42:09 (6120): No heartbeat from core client for 30 sec - exiting 14:42:10 (6120): No heartbeat from core client for 30 sec - exiting 14:42:11 (6120): No heartbeat from core client for 30 sec - exiting 14:42:12 (6120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:42:13 (6120): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t0z1ko.pjf0c10 Error converting file to netcdf: dataout/t0z1ko.pif0c10 Error converting file to netcdf: dataout/t0z1ko.pff0c10 Error converting file to netcdf: dataout/t0z1ko.pcf0c10 Error converting file to netcdf: dataout/t0z1ko.pbf0c10 Error converting file to netcdf: dataout/t0z1ko.paf0c10 Error converting file to netcdf: dataout/t0z1ka.phf0c10 Error converting file to netcdf: dataout/t0z1ka.pgf0c10 Error converting file to netcdf: dataout/t0z1ka.pef0c10 Error converting file to netcdf: dataout/t0z1ka.pdf0c10 </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Jun 2012 02:59:38	1167540	13347395	hadcm3n_t0z1_1940_40_007442812_1	259,200	401,130	1.5476
15 Oct 2011 14:22:36	1167540	13347395	hadcm3n_t0z1_1940_40_007442812_1	233,280	361,545	1.5498
15 Oct 2011 02:47:09	1167540	13347395	hadcm3n_t0z1_1940_40_007442812_1	207,360	321,659	1.5512
30 Sep 2011 14:41:25	1167540	13347395	hadcm3n_t0z1_1940_40_007442812_1	181,440	281,386	1.5508
21 Sep 2011 18:57:24	1167540	13347395	hadcm3n_t0z1_1940_40_007442812_1	155,520	241,777	1.5546
20 Sep 2011 20:02:11	1167540	13347395	hadcm3n_t0z1_1940_40_007442812_1	129,600	200,470	1.5468
19 Sep 2011 18:10:04	1167540	13347395	hadcm3n_t0z1_1940_40_007442812_1	103,680	159,884	1.5421
17 Sep 2011 03:47:57	1167540	13347395	hadcm3n_t0z1_1940_40_007442812_1	77,760	119,644	1.5386
16 Sep 2011 09:33:59	1167540	13347395	hadcm3n_t0z1_1940_40_007442812_1	51,840	79,043	1.5247
14 Sep 2011 20:21:05	1167540	13347395	hadcm3n_t0z1_1940_40_007442812_1	25,920	38,717	1.4937