Task 18552757

Name	hadam3p_anz_a3cy_2007_1_009842186_1
Workunit	9894939
Created	16 Jun 2015, 22:12:10 UTC
Sent	16 Jun 2015, 22:12:17 UTC
Report deadline	29 May 2016, 3:32:17 UTC
Received	6 Jul 2015, 22:30:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1348678
Run time	5 days 1 hours 59 min 5 sec
CPU time	3 days 18 hours 28 min 16 sec
Validate state	Invalid
Credit	2,993.82
Device peak FLOPS	3.19 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.2.47</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3008, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3784, iMonCtr=2 Model crash detected, will try to restart... 18:09:15 (4144): No heartbeat from core client for 30 sec - exiting 18:09:16 (4144): No heartbeat from core client for 30 sec - exiting 18:09:17 (4144): No heartbeat from core client for 30 sec - exiting 18:09:18 (4144): No heartbeat from core client for 30 sec - exiting 18:09:19 (4144): No heartbeat from core client for 30 sec - exiting 18:09:20 (4144): No heartbeat from core client for 30 sec - exiting 18:09:21 (4144): No heartbeat from core client for 30 sec - exiting 18:09:22 (4144): No heartbeat from core client for 30 sec - exiting 18:09:23 (4144): No heartbeat from core client for 30 sec - exiting 18:09:24 (4144): No heartbeat from core client for 30 sec - exiting 18:09:25 (4144): No heartbeat from core client for 30 sec - exiting 18:09:26 (4144): No heartbeat from core client for 30 sec - exiting 18:09:27 (4144): No heartbeat from core client for 30 sec - exiting 18:09:28 (4144): No heartbeat from core client for 30 sec - exiting 18:09:29 (4144): No heartbeat from core client for 30 sec - exiting 18:09:30 (4144): No heartbeat from core client for 30 sec - exiting 18:09:31 (4144): No heartbeat from core client for 30 sec - exiting 18:09:32 (4144): No heartbeat from core client for 30 sec - exiting 18:09:33 (4144): No heartbeat from core client for 30 sec - exiting 18:09:34 (4144): No heartbeat from core client for 30 sec - exiting 18:09:35 (4144): No heartbeat from core client for 30 sec - exiting 18:09:36 (4144): No heartbeat from core client for 30 sec - exiting 18:09:37 (4144): No heartbeat from core client for 30 sec - exiting 18:09:38 (4144): No heartbeat from core client for 30 sec - exiting 18:09:39 (4144): No heartbeat from core client for 30 sec - exiting 18:09:40 (4144): No heartbeat from core client for 30 sec - exiting 18:09:41 (4144): No heartbeat from core client for 30 sec - exiting 18:09:42 (4144): No heartbeat from core client for 30 sec - exiting 18:09:43 (4144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5408, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2968, selfPID=5148, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3364, iMonCtr=2 Model crash detected, will try to restart... 14:11:42 (5992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:00:22 (3608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6344, selfPID=6548, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 18:08:23 (4272): No heartbeat from core client for 30 sec - exiting 18:08:24 (4272): No heartbeat from core client for 30 sec - exiting 18:08:25 (4272): No heartbeat from core client for 30 sec - exiting 18:08:26 (4272): No heartbeat from core client for 30 sec - exiting 18:08:27 (4272): No heartbeat from core client for 30 sec - exiting 18:08:28 (4272): No heartbeat from core client for 30 sec - exiting 18:08:29 (4272): No heartbeat from core client for 30 sec - exiting 18:08:30 (4272): No heartbeat from core client for 30 sec - exiting 18:08:31 (4272): No heartbeat from core client for 30 sec - exiting 18:08:32 (4272): No heartbeat from core client for 30 sec - exiting 18:08:33 (4272): No heartbeat from core client for 30 sec - exiting 18:08:34 (4272): No heartbeat from core client for 30 sec - exiting 18:08:35 (4272): No heartbeat from core client for 30 sec - exiting 18:08:36 (4272): No heartbeat from core client for 30 sec - exiting 18:08:37 (4272): No heartbeat from core client for 30 sec - exiting 18:08:38 (4272): No heartbeat from core client for 30 sec - exiting 18:08:39 (4272): No heartbeat from core client for 30 sec - exiting 18:08:40 (4272): No heartbeat from core client for 30 sec - exiting 18:08:41 (4272): No heartbeat from core client for 30 sec - exiting 18:08:42 (4272): No heartbeat from core client for 30 sec - exiting 18:08:43 (4272): No heartbeat from core client for 30 sec - exiting 18:08:44 (4272): No heartbeat from core client for 30 sec - exiting 18:08:45 (4272): No heartbeat from core client for 30 sec - exiting 18:08:46 (4272): No heartbeat from core client for 30 sec - exiting 18:08:47 (4272): No heartbeat from core client for 30 sec - exiting 18:08:48 (4272): No heartbeat from core client for 30 sec - exiting 18:08:49 (4272): No heartbeat from core client for 30 sec - exiting 18:08:50 (4272): No heartbeat from core client for 30 sec - exiting 18:08:51 (4272): No heartbeat from core client for 30 sec - exiting 18:08:52 (4272): No heartbeat from core client for 30 sec - exiting 18:08:53 (4272): No heartbeat from core client for 30 sec - exiting 18:08:54 (4272): No heartbeat from core client for 30 sec - exiting 18:08:55 (4272): No heartbeat from core client for 30 sec - exiting 18:08:56 (4272): No heartbeat from core client for 30 sec - exiting 18:08:57 (4272): No heartbeat from core client for 30 sec - exiting 18:08:58 (4272): No heartbeat from core client for 30 sec - exiting 18:08:59 (4272): No heartbeat from core client for 30 sec - exiting 18:09:00 (4272): No heartbeat from core client for 30 sec - exiting 18:09:01 (4272): No heartbeat from core client for 30 sec - exiting 18:09:02 (4272): No heartbeat from core client for 30 sec - exiting 18:09:03 (4272): No heartbeat from core client for 30 sec - exiting 18:09:04 (4272): No heartbeat from core client for 30 sec - exiting 18:09:05 (4272): No heartbeat from core client for 30 sec - exiting 18:09:06 (4272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:14 (3256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5824, selfPID=5824, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=708, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 18:02:42 (2904): No heartbeat from core client for 30 sec - exiting 18:02:43 (2904): No heartbeat from core client for 30 sec - exiting 18:02:44 (2904): No heartbeat from core client for 30 sec - exiting 18:02:45 (2904): No heartbeat from core client for 30 sec - exiting 18:02:46 (2904): No heartbeat from core client for 30 sec - exiting 18:02:47 (2904): No heartbeat from core client for 30 sec - exiting 18:02:48 (2904): No heartbeat from core client for 30 sec - exiting 18:02:49 (2904): No heartbeat from core client for 30 sec - exiting 18:02:50 (2904): No heartbeat from core client for 30 sec - exiting 18:02:51 (2904): No heartbeat from core client for 30 sec - exiting 18:02:52 (2904): No heartbeat from core client for 30 sec - exiting 18:02:53 (2904): No heartbeat from core client for 30 sec - exiting 18:02:54 (2904): No heartbeat from core client for 30 sec - exiting 18:02:55 (2904): No heartbeat from core client for 30 sec - exiting 18:02:56 (2904): No heartbeat from core client for 30 sec - exiting 18:02:57 (2904): No heartbeat from core client for 30 sec - exiting 18:02:58 (2904): No heartbeat from core client for 30 sec - exiting 18:02:59 (2904): No heartbeat from core client for 30 sec - exiting 18:03:00 (2904): No heartbeat from core client for 30 sec - exiting 18:03:01 (2904): No heartbeat from core client for 30 sec - exiting 18:03:02 (2904): No heartbeat from core client for 30 sec - exiting 18:03:03 (2904): No heartbeat from core client for 30 sec - exiting 18:03:04 (2904): No heartbeat from core client for 30 sec - exiting 18:03:05 (2904): No heartbeat from core client for 30 sec - exiting 18:03:06 (2904): No heartbeat from core client for 30 sec - exiting 18:03:07 (2904): No heartbeat from core client for 30 sec - exiting 18:03:08 (2904): No heartbeat from core client for 30 sec - exiting 18:03:09 (2904): No heartbeat from core client for 30 sec - exiting 18:03:10 (2904): No heartbeat from core client for 30 sec - exiting 18:03:11 (2904): No heartbeat from core client for 30 sec - exiting 18:03:12 (2904): No heartbeat from core client for 30 sec - exiting 18:03:13 (2904): No heartbeat from core client for 30 sec - exiting 18:03:14 (2904): No heartbeat from core client for 30 sec - exiting 18:03:15 (2904): No heartbeat from core client for 30 sec - exiting 18:03:16 (2904): No heartbeat from core client for 30 sec - exiting 18:03:17 (2904): No heartbeat from core client for 30 sec - exiting 18:03:18 (2904): No heartbeat from core client for 30 sec - exiting 18:03:19 (2904): No heartbeat from core client for 30 sec - exiting 18:03:20 (2904): No heartbeat from core client for 30 sec - exiting 18:03:21 (2904): No heartbeat from core client for 30 sec - exiting 18:03:22 (2904): No heartbeat from core client for 30 sec - exiting 18:03:23 (2904): No heartbeat from core client for 30 sec - exiting 18:03:24 (2904): No heartbeat from core client for 30 sec - exiting 18:03:25 (2904): No heartbeat from core client for 30 sec - exiting 18:03:26 (2904): No heartbeat from core client for 30 sec - exiting 18:03:27 (2904): No heartbeat from core client for 30 sec - exiting 18:03:28 (2904): No heartbeat from core client for 30 sec - exiting 18:03:29 (2904): No heartbeat from core client for 30 sec - exiting 18:03:30 (2904): No heartbeat from core client for 30 sec - exiting 18:03:31 (2904): No heartbeat from core client for 30 sec - exiting 18:03:32 (2904): No heartbeat from core client for 30 sec - exiting 18:03:33 (2904): No heartbeat from core client for 30 sec - exiting 18:03:34 (2904): No heartbeat from core client for 30 sec - exiting 18:03:35 (2904): No heartbeat from core client for 30 sec - exiting 18:03:36 (2904): No heartbeat from core client for 30 sec - exiting 18:03:37 (2904): No heartbeat from core client for 30 sec - exiting 18:03:38 (2904): No heartbeat from core client for 30 sec - exiting 18:03:39 (2904): No heartbeat from core client for 30 sec - exiting 18:03:40 (2904): No heartbeat from core client for 30 sec - exiting 18:03:41 (2904): No heartbeat from core client for 30 sec - exiting 18:03:42 (2904): No heartbeat from core client for 30 sec - exiting 18:03:43 (2904): No heartbeat from core client for 30 sec - exiting 18:03:44 (2904): No heartbeat from core client for 30 sec - exiting 18:03:45 (2904): No heartbeat from core client for 30 sec - exiting 18:03:46 (2904): No heartbeat from core client for 30 sec - exiting 18:03:47 (2904): No heartbeat from core client for 30 sec - exiting 18:03:48 (2904): No heartbeat from core client for 30 sec - exiting 18:03:49 (2904): No heartbeat from core client for 30 sec - exiting 18:03:50 (2904): No heartbeat from core client for 30 sec - exiting 18:03:51 (2904): No heartbeat from core client for 30 sec - exiting 18:03:52 (2904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5316, selfPID=1040, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... GController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5496, selfPID=4072, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 18:04:17 (1512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5476, selfPID=5476, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4332, selfPID=5360, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 15:01:37 (5444): No heartbeat from core client for 30 sec - exiting 15:01:38 (5444): No heartbeat from core client for 30 sec - exiting 15:01:39 (5444): No heartbeat from core client for 30 sec - exiting 15:01:40 (5444): No heartbeat from core client for 30 sec - exiting 15:01:41 (5444): No heartbeat from core client for 30 sec - exiting 15:01:42 (5444): No heartbeat from core client for 30 sec - exiting 15:01:43 (5444): No heartbeat from core client for 30 sec - exiting 15:01:45 (5444): No heartbeat from core client for 30 sec - exiting 15:01:46 (5444): No heartbeat from core client for 30 sec - exiting 15:01:47 (5444): No heartbeat from core client for 30 sec - exiting 15:01:48 (5444): No heartbeat from core client for 30 sec - exiting 15:01:49 (5444): No heartbeat from core client for 30 sec - exiting 15:01:50 (5444): No heartbeat from core client for 30 sec - exiting 15:01:51 (5444): No heartbeat from core client for 30 sec - exiting 15:01:52 (5444): No heartbeat from core client for 30 sec - exiting 15:01:53 (5444): No heartbeat from core client for 30 sec - exiting 15:01:54 (5444): No heartbeat from core client for 30 sec - exiting 15:01:56 (5444): No heartbeat from core client for 30 sec - exiting 15:01:57 (5444): No heartbeat from core client for 30 sec - exiting 15:01:58 (5444): No heartbeat from core client for 30 sec - exiting 15:01:59 (5444): No heartbeat from core client for 30 sec - exiting 15:02:00 (5444): No heartbeat from core client for 30 sec - exiting 15:02:01 (5444): No heartbeat from core client for 30 sec - exiting 15:02:02 (5444): No heartbeat from core client for 30 sec - exiting 15:02:03 (5444): No heartbeat from core client for 30 sec - exiting 15:02:04 (5444): No heartbeat from core client for 30 sec - exiting 15:02:05 (5444): No heartbeat from core client for 30 sec - exiting 15:02:06 (5444): No heartbeat from core client for 30 sec - exiting 15:02:08 (5444): No heartbeat from core client for 30 sec - exiting 15:02:09 (5444): No heartbeat from core client for 30 sec - exiting 15:02:10 (5444): No heartbeat from core client for 30 sec - exiting 15:02:11 (5444): No heartbeat from core client for 30 sec - exiting 15:02:12 (5444): No heartbeat from core client for 30 sec - exiting 15:02:13 (5444): No heartbeat from core client for 30 sec - exiting 15:02:14 (5444): No heartbeat from core client for 30 sec - exiting 15:02:15 (5444): No heartbeat from core client for 30 sec - exiting 15:02:16 (5444): No heartbeat from core client for 30 sec - exiting 15:02:17 (5444): No heartbeat from core client for 30 sec - exiting 15:02:18 (5444): No heartbeat from core client for 30 sec - exiting 15:02:20 (5444): No heartbeat from core client for 30 sec - exiting 15:02:21 (5444): No heartbeat from core client for 30 sec - exiting 15:02:22 (5444): No heartbeat from core client for 30 sec - exiting 15:02:23 (5444): No heartbeat from core client for 30 sec - exiting 15:02:24 (5444): No heartbeat from core client for 30 sec - exiting 15:02:25 (5444): No heartbeat from core client for 30 sec - exiting 15:02:26 (5444): No heartbeat from core client for 30 sec - exiting 15:02:27 (5444): No heartbeat from core client for 30 sec - exiting 15:02:28 (5444): No heartbeat from core client for 30 sec - exiting 15:02:29 (5444): No heartbeat from core client for 30 sec - exiting 15:02:30 (5444): No heartbeat from core client for 30 sec - exiting 15:02:32 (5444): No heartbeat from core client for 30 sec - exiting 15:02:33 (5444): No heartbeat from core client for 30 sec - exiting 15:02:34 (5444): No heartbeat from core client for 30 sec - exiting 15:02:35 (5444): No heartbeat from core client for 30 sec - exiting 15:02:36 (5444): No heartbeat from core client for 30 sec - exiting 15:02:37 (5444): No heartbeat from core client for 30 sec - exiting 15:02:38 (5444): No heartbeat from core client for 30 sec - exiting 15:02:39 (5444): No heartbeat from core client for 30 sec - exiting 15:02:40 (5444): No heartbeat from core client for 30 sec - exiting 15:02:41 (5444): No heartbeat from core client for 30 sec - exiting 15:02:43 (5444): No heartbeat from core client for 30 sec - exiting 15:02:44 (5444): No heartbeat from core client for 30 sec - exiting 15:02:45 (5444): No heartbeat from core client for 30 sec - exiting 15:02:46 (5444): No heartbeat from core client for 30 sec - exiting 15:02:47 (5444): No heartbeat from core client for 30 sec - exiting 15:02:48 (5444): No heartbeat from core client for 30 sec - exiting 15:02:49 (5444): No heartbeat from core client for 30 sec - exiting 15:02:50 (5444): No heartbeat from core client for 30 sec - exiting 15:02:51 (5444): No heartbeat from core client for 30 sec - exiting 15:02:52 (5444): No heartbeat from core client for 30 sec - exiting 15:02:53 (5444): No heartbeat from core client for 30 sec - exiting 15:02:55 (5444): No heartbeat from core client for 30 sec - exiting 15:02:56 (5444): No heartbeat from core client for 30 sec - exiting 15:02:57 (5444): No heartbeat from core client for 30 sec - exiting 15:02:58 (5444): No heartbeat from core client for 30 sec - exiting 15:02:59 (5444): No heartbeat from core client for 30 sec - exiting 15:03:00 (5444): No heartbeat from core client for 30 sec - exiting 15:03:01 (5444): No heartbeat from core client for 30 sec - exiting 15:03:02 (5444): No heartbeat from core client for 30 sec - exiting 15:03:03 (5444): No heartbeat from core client for 30 sec - exiting 15:03:04 (5444): No heartbeat from core client for 30 sec - exiting 15:03:05 (5444): No heartbeat from core client for 30 sec - exiting 15:03:07 (5444): No heartbeat from core client for 30 sec - exiting 15:03:08 (5444): No heartbeat from core client for 30 sec - exiting 15:03:09 (5444): No heartbeat from core client for 30 sec - exiting 15:03:10 (5444): No heartbeat from core client for 30 sec - exiting 15:03:11 (5444): No heartbeat from core client for 30 sec - exiting 15:03:12 (5444): No heartbeat from core client for 30 sec - exiting 15:03:13 (5444): No heartbeat from core client for 30 sec - exiting 15:03:14 (5444): No heartbeat from core client for 30 sec - exiting 15:03:15 (5444): No heartbeat from core client for 30 sec - exiting 15:03:16 (5444): No heartbeat from core client for 30 sec - exiting 15:03:17 (5444): No heartbeat from core client for 30 sec - exiting 15:03:19 (5444): No heartbeat from core client for 30 sec - exiting 15:03:20 (5444): No heartbeat from core client for 30 sec - exiting 15:03:21 (5444): No heartbeat from core client for 30 sec - exiting 15:03:22 (5444): No heartbeat from core client for 30 sec - exiting 15:03:23 (5444): No heartbeat from core client for 30 sec - exiting 15:03:24 (5444): No heartbeat from core client for 30 sec - exiting 15:03:25 (5444): No heartbeat from core client for 30 sec - exiting 15:03:26 (5444): No heartbeat from core client for 30 sec - exiting 15:03:27 (5444): No heartbeat from core client for 30 sec - exiting 15:03:28 (5444): No heartbeat from core client for 30 sec - exiting 15:03:29 (5444): No heartbeat from core client for 30 sec - exiting 15:03:31 (5444): No heartbeat from core client for 30 sec - exiting 15:03:32 (5444): No heartbeat from core client for 30 sec - exiting 15:03:33 (5444): No heartbeat from core client for 30 sec - exiting 15:03:34 (5444): No heartbeat from core client for 30 sec - exiting 15:03:35 (5444): No heartbeat from core client for 30 sec - exiting 15:03:36 (5444): No heartbeat from core client for 30 sec - exiting 15:03:37 (5444): No heartbeat from core client for 30 sec - exiting 15:03:38 (5444): No heartbeat from core client for 30 sec - exiting 15:03:39 (5444): No heartbeat from core client for 30 sec - exiting 15:03:40 (5444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5280, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5972, iMonCtr=2 13:25:48 (2988): No heartbeat from core client for 30 sec - exiting 13:25:49 (2988): No heartbeat from core client for 30 sec - exiting 13:25:50 (2988): No heartbeat from core client for 30 sec - exiting 13:25:51 (2988): No heartbeat from core client for 30 sec - exiting 13:25:52 (2988): No heartbeat from core client for 30 sec - exiting 13:25:53 (2988): No heartbeat from core client for 30 sec - exiting 13:25:54 (2988): No heartbeat from core client for 30 sec - exiting 13:25:55 (2988): No heartbeat from core client for 30 sec - exiting 13:25:57 (2988): No heartbeat from core client for 30 sec - exiting 13:25:58 (2988): No heartbeat from core client for 30 sec - exiting 13:25:59 (2988): No heartbeat from core client for 30 sec - exiting 13:26:00 (2988): No heartbeat from core client for 30 sec - exiting 13:26:01 (2988): No heartbeat from core client for 30 sec - exiting 13:26:02 (2988): No heartbeat from core client for 30 sec - exiting 13:26:03 (2988): No heartbeat from core client for 30 sec - exiting 13:26:04 (2988): No heartbeat from core client for 30 sec - exiting 13:26:05 (2988): No heartbeat from core client for 30 sec - exiting 13:26:06 (2988): No heartbeat from core client for 30 sec - exiting 13:26:07 (2988): No heartbeat from core client for 30 sec - exiting 13:26:09 (2988): No heartbeat from core client for 30 sec - exiting 13:26:10 (2988): No heartbeat from core client for 30 sec - exiting 13:26:11 (2988): No heartbeat from core client for 30 sec - exiting 13:26:12 (2988): No heartbeat from core client for 30 sec - exiting 13:26:13 (2988): No heartbeat from core client for 30 sec - exiting 13:26:14 (2988): No heartbeat from core client for 30 sec - exiting 13:26:15 (2988): No heartbeat from core client for 30 sec - exiting 13:26:16 (2988): No heartbeat from core client for 30 sec - exiting 13:26:17 (2988): No heartbeat from core client for 30 sec - exiting 13:26:18 (2988): No heartbeat from core client for 30 sec - exiting 13:26:19 (2988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1360, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4812, selfPID=5284, iMonCtr=1 Model crash detected, will try to restart... 18:00:08 (3496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=776, selfPID=776, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... 18:07:38 (3920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:39 (3920): No heartbeat from core client for 30 sec - exiting 18:07:40 (3920): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:22:23 (2964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:22:30 (2964): No heartbeat from core client for 30 sec - exiting 00:22:31 (2964): No heartbeat from core client for 30 sec - exiting Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5012, selfPID=5012, iMonCtr=2 00:22:32 (2964): No heartbeat from core client for 30 sec - exiting 00:22:34 (2964): No heartbeat from core client for 30 sec - exiting 00:22:35 (2964): No heartbeat from core client for 30 sec - exiting 00:22:36 (2964): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:12:38 (5880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... GController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7704, selfPID=7956, iMonCtr=1 Model crash detected, will try to restart... 18:20:19 (5416): No heartbeat from core client for 30 sec - exiting 18:20:20 (5416): No heartbeat from core client for 30 sec - exiting 18:20:21 (5416): No heartbeat from core client for 30 sec - exiting 18:20:22 (5416): No heartbeat from core client for 30 sec - exiting 18:20:23 (5416): No heartbeat from core client for 30 sec - exiting 18:20:24 (5416): No heartbeat from core client for 30 sec - exiting 18:20:25 (5416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3020, selfPID=3020, iMonCtr=2 18:20:26 (5416): No heartbeat from core client for 30 sec - exiting Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2656, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3376, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 13:39:19 (3464): No heartbeat from core client for 30 sec - exiting 13:39:20 (3464): No heartbeat from core client for 30 sec - exiting 13:39:21 (3464): No heartbeat from core client for 30 sec - exiting 13:39:22 (3464): No heartbeat from core client for 30 sec - exiting 13:39:23 (3464): No heartbeat from core client for 30 sec - exiting 13:39:24 (3464): No heartbeat from core client for 30 sec - exiting 13:39:25 (3464): No heartbeat from core client for 30 sec - exiting 13:39:27 (3464): No heartbeat from core client for 30 sec - exiting 13:39:28 (3464): No heartbeat from core client for 30 sec - exiting 13:39:29 (3464): No heartbeat from core client for 30 sec - exiting 13:39:30 (3464): No heartbeat from core client for 30 sec - exiting 13:39:31 (3464): No heartbeat from core client for 30 sec - exiting 13:39:32 (3464): No heartbeat from core client for 30 sec - exiting 13:39:33 (3464): No heartbeat from core client for 30 sec - exiting 13:39:34 (3464): No heartbeat from core client for 30 sec - exiting 13:39:35 (3464): No heartbeat from core client for 30 sec - exiting 13:39:36 (3464): No heartbeat from core client for 30 sec - exiting 13:39:37 (3464): No heartbeat from core client for 30 sec - exiting 13:39:39 (3464): No heartbeat from core client for 30 sec - exiting 13:39:40 (3464): No heartbeat from core client for 30 sec - exiting 13:39:41 (3464): No heartbeat from core client for 30 sec - exiting 13:39:42 (3464): No heartbeat from core client for 30 sec - exiting 13:39:43 (3464): No heartbeat from core client for 30 sec - exiting 13:39:44 (3464): No heartbeat from core client for 30 sec - exiting 13:39:45 (3464): No heartbeat from core client for 30 sec - exiting 13:39:46 (3464): No heartbeat from core client for 30 sec - exiting 13:39:47 (3464): No heartbeat from core client for 30 sec - exiting 13:39:48 (3464): No heartbeat from core client for 30 sec - exiting 13:39:50 (3464): No heartbeat from core client for 30 sec - exiting 13:39:51 (3464): No heartbeat from core client for 30 sec - exiting 13:39:52 (3464): No heartbeat from core client for 30 sec - exiting 13:39:53 (3464): No heartbeat from core client for 30 sec - exiting 13:39:54 (3464): No heartbeat from core client for 30 sec - exiting 13:39:55 (3464): No heartbeat from core client for 30 sec - exiting 13:39:56 (3464): No heartbeat from core client for 30 sec - exiting 13:39:57 (3464): No heartbeat from core client for 30 sec - exiting 13:39:58 (3464): No heartbeat from core client for 30 sec - exiting 13:39:59 (3464): No heartbeat from core client for 30 sec - exiting 13:40:00 (3464): No heartbeat from core client for 30 sec - exiting 13:40:02 (3464): No heartbeat from core client for 30 sec - exiting 13:40:03 (3464): No heartbeat from core client for 30 sec - exiting 13:40:04 (3464): No heartbeat from core client for 30 sec - exiting 13:40:05 (3464): No heartbeat from core client for 30 sec - exiting 13:40:06 (3464): No heartbeat from core client for 30 sec - exiting 13:40:07 (3464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3740, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6108, selfPID=5336, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3704, selfPID=3396, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6048, selfPID=1176, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 18:02:02 (4452): No heartbeat from core client for 30 sec - exiting 18:02:03 (4452): No heartbeat from core client for 30 sec - exiting 18:02:04 (4452): No heartbeat from core client for 30 sec - exiting 18:02:05 (4452): No heartbeat from core client for 30 sec - exiting 18:02:06 (4452): No heartbeat from core client for 30 sec - exiting 18:02:07 (4452): No heartbeat from core client for 30 sec - exiting 18:02:08 (4452): No heartbeat from core client for 30 sec - exiting 18:02:09 (4452): No heartbeat from core client for 30 sec - exiting 18:02:10 (4452): No heartbeat from core client for 30 sec - exiting 18:02:11 (4452): No heartbeat from core client for 30 sec - exiting 18:02:12 (4452): No heartbeat from core client for 30 sec - exiting 18:02:13 (4452): No heartbeat from core client for 30 sec - exiting 18:02:14 (4452): No heartbeat from core client for 30 sec - exiting 18:02:15 (4452): No heartbeat from core client for 30 sec - exiting 18:02:16 (4452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_a3cy_2007_1_009842186_1_7.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_a3cy_2007_1_009842186_1_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_a3cy_2007_1_009842186_1_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_a3cy_2007_1_009842186_1_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_a3cy_2007_1_009842186_1_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_a3cy_2007_1_009842186_1_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Jul 2015 19:49:20	1348678	18552757	hadam3p_anz_a3cy_2007_1_009842186_1	69,419	305,246	4.3972
04 Jul 2015 00:43:41	1348678	18552757	hadam3p_anz_a3cy_2007_1_009842186_1	57,899	254,399	4.3938
01 Jul 2015 06:27:39	1348678	18552757	hadam3p_anz_a3cy_2007_1_009842186_1	46,379	204,309	4.4052
28 Jun 2015 22:24:16	1348678	18552757	hadam3p_anz_a3cy_2007_1_009842186_1	34,859	154,068	4.4197
26 Jun 2015 02:22:55	1348678	18552757	hadam3p_anz_a3cy_2007_1_009842186_1	23,339	103,552	4.4369
21 Jun 2015 03:55:10	1348678	18552757	hadam3p_anz_a3cy_2007_1_009842186_1	11,819	53,137	4.4959