Task 15455112

Name	hadam3p_pnw_761z_2004_1_007603087_1
Workunit	7781217
Created	22 Nov 2012, 19:24:27 UTC
Sent	22 Nov 2012, 19:25:03 UTC
Report deadline	5 Nov 2013, 0:45:03 UTC
Received	10 Apr 2013, 21:15:55 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1091233
Run time	27 days 12 hours 58 min 26 sec
CPU time	23 days 23 hours 13 min 22 sec
Validate state	Invalid
Credit	2,755.56
Device peak FLOPS	0.69 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Pacific North West v6.09 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3600, selfPID=2140, iMonCtr=1 Model crash detected, CPDN Monitor - Quit request from BOINC... Regional WorkerCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1208, selfPID=4516, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5564, selfPID=5564, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2416, selfPID=2416, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5332, selfPID=5332, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6604, selfPID=2784, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2440, selfPID=5408, iMonCtr=1 Model crash detected, will try to restart... 08:45:15 (5320): No heartbeat from core client for 30 sec - exiting 08:45:16 (5320): No heartbeat from core client for 30 sec - exiting 08:45:17 (5320): No heartbeat from core client for 30 sec - exiting 08:45:19 (5320): No heartbeat from core client for 30 sec - exiting 08:45:20 (5320): No heartbeat from core client for 30 sec - exiting 08:45:21 (5320): No heartbeat from core client for 30 sec - exiting 08:45:23 (5320): No heartbeat from core client for 30 sec - exiting 08:45:24 (5320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... RegCPDN Monitor - Quit request from BOINC... 09:33:00 (5996): No heartbeat from core client for 30 sec - exiting 09:33:02 (5996): No heartbeat from core client for 30 sec - exiting 09:33:03 (5996): No heartbeat from core client for 30 sec - exiting 09:33:05 (5996): No heartbeat from core client for 30 sec - exiting 09:33:06 (5996): No heartbeat from core client for 30 sec - exiting 09:33:07 (5996): No heartbeat from core client for 30 sec - exiting 09:33:08 (5996): No heartbeat from core client for 30 sec - exiting 09:33:09 (5996): No heartbeat from core client for 30 sec - exiting 09:33:10 (5996): No heartbeat from core client for 30 sec - exiting 09:33:11 (5996): No heartbeat from core client for 30 sec - exiting 09:33:12 (5996): No heartbeat from core client for 30 sec - exiting 09:33:13 (5996): No heartbeat from core client for 30 sec - exiting 09:33:14 (5996): No heartbeat from core client for 30 sec - exiting 09:33:15 (5996): No heartbeat from core client for 30 sec - exiting 09:33:16 (5996): No heartbeat from core client for 30 sec - exiting 09:33:17 (5996): No heartbeat from core client for 30 sec - exiting 09:33:18 (5996): No heartbeat from core client for 30 sec - exiting 09:33:19 (5996): No heartbeat from core client for 30 sec - exiting 09:33:20 (5996): No heartbeat from core client for 30 sec - exiting 09:33:21 (5996): No heartbeat from core client for 30 sec - exiting 09:33:22 (5996): No heartbeat from core client for 30 sec - exiting 09:33:23 (5996): No heartbeat from core client for 30 sec - exiting 09:33:24 (5996): No heartbeat from core client for 30 sec - exiting 09:33:25 (5996): No heartbeat from core client for 30 sec - exiting 09:33:26 (5996): No heartbeat from core client for 30 sec - exiting 09:33:27 (5996): No heartbeat from core client for 30 sec - exiting 09:33:28 (5996): No heartbeat from core client for 30 sec - exiting 09:33:29 (5996): No heartbeat from core client for 30 sec - exiting 09:33:30 (5996): No heartbeat from core client for 30 sec - exiting 09:33:31 (5996): No heartbeat from core client for 30 sec - exiting 09:33:32 (5996): No heartbeat from core client for 30 sec - exiting 09:33:33 (5996): No heartbeat from core client for 30 sec - exiting 09:33:34 (5996): No heartbeat from core client for 30 sec - exiting 09:33:35 (5996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:02:22 (2532): No heartbeat from core client for 30 sec - exiting 10:02:23 (2532): No heartbeat from core client for 30 sec - exiting 10:02:24 (2532): No heartbeat from core client for 30 sec - exiting 10:02:25 (2532): No heartbeat from core client for 30 sec - exiting 10:02:26 (2532): No heartbeat from core client for 30 sec - exiting 10:02:27 (2532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2796, selfPID=2796, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2704, iMonCtr=2 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5024, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5588, selfPID=4568, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6056, selfPID=5124, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5076, selfPID=5076, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5180, selfPID=5180, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... RegionCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3004, iMonCtr=2 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4820, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6672, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7684, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8336, selfPID=8768, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1940, selfPID=1940, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:54:19 (6532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:57:15 (5368): No heartbeat from core client for 30 sec - exiting 06:57:17 (5368): No heartbeat from core client for 30 sec - exiting 06:57:18 (5368): No heartbeat from core client for 30 sec - exiting 06:57:19 (5368): No heartbeat from core client for 30 sec - exiting 06:57:20 (5368): No heartbeat from core client for 30 sec - exiting 06:57:22 (5368): No heartbeat from core client for 30 sec - exiting 06:57:23 (5368): No heartbeat from core client for 30 sec - exiting 06:57:24 (5368): No heartbeat from core client for 30 sec - exiting 06:57:26 (5368): No heartbeat from core client for 30 sec - exiting 06:57:27 (5368): No heartbeat from core client for 30 sec - exiting 06:57:28 (5368): No heartbeat from core client for 30 sec - exiting 06:57:29 (5368): No heartbeat from core client for 30 sec - exiting 06:57:30 (5368): No heartbeat from core client for 30 sec - exiting 06:57:31 (5368): No heartbeat from core client for 30 sec - exiting 06:57:32 (5368): No heartbeat from core client for 30 sec - exiting 06:57:33 (5368): No heartbeat from core client for 30 sec - exiting 06:57:34 (5368): No heartbeat from core client for 30 sec - exiting 06:57:35 (5368): No heartbeat from core client for 30 sec - exiting 06:57:36 (5368): No heartbeat from core client for 30 sec - exiting 06:57:37 (5368): No heartbeat from core client for 30 sec - exiting 06:57:38 (5368): No heartbeat from core client for 30 sec - exiting 06:57:39 (5368): No heartbeat from core client for 30 sec - exiting 06:57:40 (5368): No heartbeat from core client for 30 sec - exiting 06:57:42 (5368): No heartbeat from core client for 30 sec - exiting 06:57:43 (5368): No heartbeat from core client for 30 sec - exiting 06:57:44 (5368): No heartbeat from core client for 30 sec - exiting 06:57:45 (5368): No heartbeat from core client for 30 sec - exiting 06:57:46 (5368): No heartbeat from core client for 30 sec - exiting 06:57:47 (5368): No heartbeat from core client for 30 sec - exiting 06:57:48 (5368): No heartbeat from core client for 30 sec - exiting 06:57:49 (5368): No heartbeat from core client for 30 sec - exiting 06:57:50 (5368): No heartbeat from core client for 30 sec - exiting 06:57:52 (5368): No heartbeat from core client for 30 sec - exiting 06:57:53 (5368): No heartbeat from core client for 30 sec - exiting 06:57:54 (5368): No heartbeat from core client for 30 sec - exiting 06:57:55 (5368): No heartbeat from core client for 30 sec - exiting 06:57:56 (5368): No heartbeat from core client for 30 sec - exiting 06:57:57 (5368): No heartbeat from core client for 30 sec - exiting 06:57:58 (5368): No heartbeat from core client for 30 sec - exiting 06:57:59 (5368): No heartbeat from core client for 30 sec - exiting 06:58:00 (5368): No heartbeat from core client for 30 sec - exiting 06:58:01 (5368): No heartbeat from core client for 30 sec - exiting 06:58:02 (5368): No heartbeat from core client for 30 sec - exiting 06:58:03 (5368): No heartbeat from core client for 30 sec - exiting 06:58:04 (5368): No heartbeat from core client for 30 sec - exiting 06:58:05 (5368): No heartbeat from core client for 30 sec - exiting 06:58:06 (5368): No heartbeat from core client for 30 sec - exiting 06:58:07 (5368): No heartbeat from core client for 30 sec - exiting 06:58:08 (5368): No heartbeat from core client for 30 sec - exiting 06:58:10 (5368): No heartbeat from core client for 30 sec - exiting 06:58:11 (5368): No heartbeat from core client for 30 sec - exiting 06:58:12 (5368): No heartbeat from core client for 30 sec - exiting 06:58:14 (5368): No heartbeat from core client for 30 sec - exiting 06:58:15 (5368): No heartbeat from core client for 30 sec - exiting 06:58:16 (5368): No heartbeat from core client for 30 sec - exiting 06:58:17 (5368): No heartbeat from core client for 30 sec - exiting 06:58:18 (5368): No heartbeat from core client for 30 sec - exiting 06:58:19 (5368): No heartbeat from core client for 30 sec - exiting 06:58:20 (5368): No heartbeat from core client for 30 sec - exiting 06:58:21 (5368): No heartbeat from core client for 30 sec - exiting 06:58:22 (5368): No heartbeat from core client for 30 sec - exiting 06:58:23 (5368): No heartbeat from core client for 30 sec - exiting 06:58:24 (5368): No heartbeat from core client for 30 sec - exiting 06:58:25 (5368): No heartbeat from core client for 30 sec - exiting 06:58:26 (5368): No heartbeat from core client for 30 sec - exiting 06:58:27 (5368): No heartbeat from core client for 30 sec - exiting 06:58:28 (5368): No heartbeat from core client for 30 sec - exiting 06:58:29 (5368): No heartbeat from core client for 30 sec - exiting 06:58:30 (5368): No heartbeat from core client for 30 sec - exiting 06:58:31 (5368): No heartbeat from core client for 30 sec - exiting 06:58:32 (5368): No heartbeat from core client for 30 sec - exiting 06:58:33 (5368): No heartbeat from core client for 30 sec - exiting 06:58:34 (5368): No heartbeat from core client for 30 sec - exiting 06:58:35 (5368): No heartbeat from core client for 30 sec - exiting 06:58:36 (5368): No heartbeat from core client for 30 sec - exiting 06:58:37 (5368): No heartbeat from core client for 30 sec - exiting 06:58:38 (5368): No heartbeat from core client for 30 sec - exiting 06:58:39 (5368): No heartbeat from core client for 30 sec - exiting 06:58:40 (5368): No heartbeat from core client for 30 sec - exiting 06:58:41 (5368): No heartbeat from core client for 30 sec - exiting 06:58:42 (5368): No heartbeat from core client for 30 sec - exiting 06:58:43 (5368): No heartbeat from core client for 30 sec - exiting 06:58:44 (5368): No heartbeat from core client for 30 sec - exiting 06:58:45 (5368): No heartbeat from core client for 30 sec - exiting 06:58:46 (5368): No heartbeat from core client for 30 sec - exiting 06:58:47 (5368): No heartbeat from core client for 30 sec - exiting 06:58:48 (5368): No heartbeat from core client for 30 sec - exiting 06:58:49 (5368): No heartbeat from core client for 30 sec - exiting 06:58:50 (5368): No heartbeat from core client for 30 sec - exiting 06:58:51 (5368): No heartbeat from core client for 30 sec - exiting 06:58:52 (5368): No heartbeat from core client for 30 sec - exiting 06:58:53 (5368): No heartbeat from core client for 30 sec - exiting 06:58:54 (5368): No heartbeat from core client for 30 sec - exiting 06:58:55 (5368): No heartbeat from core client for 30 sec - exiting 06:58:56 (5368): No heartbeat from core client for 30 sec - exiting 06:58:57 (5368): No heartbeat from core client for 30 sec - exiting 06:58:58 (5368): No heartbeat from core client for 30 sec - exiting 06:58:59 (5368): No heartbeat from core client for 30 sec - exiting 06:59:00 (5368): No heartbeat from core client for 30 sec - exiting 06:59:01 (5368): No heartbeat from core client for 30 sec - exiting 06:59:02 (5368): No heartbeat from core client for 30 sec - exiting 06:59:03 (5368): No heartbeat from core client for 30 sec - exiting 06:59:04 (5368): No heartbeat from core client for 30 sec - exiting 06:59:05 (5368): No heartbeat from core client for 30 sec - exiting 06:59:06 (5368): No heartbeat from core client for 30 sec - exiting 06:59:07 (5368): No heartbeat from core client for 30 sec - exiting 06:59:08 (5368): No heartbeat from core client for 30 sec - exiting 06:59:09 (5368): No heartbeat from core client for 30 sec - exiting 06:59:10 (5368): No heartbeat from core client for 30 sec - exiting 06:59:11 (5368): No heartbeat from core client for 30 sec - exiting 06:59:12 (5368): No heartbeat from core client for 30 sec - exiting 06:59:13 (5368): No heartbeat from core client for 30 sec - exiting 06:59:14 (5368): No heartbeat from core client for 30 sec - exiting 06:59:15 (5368): No heartbeat from core client for 30 sec - exiting 06:59:16 (5368): No heartbeat from core client for 30 sec - exiting 06:59:17 (5368): No heartbeat from core client for 30 sec - exiting 06:59:18 (5368): No heartbeat from core client for 30 sec - exiting 06:59:19 (5368): No heartbeat from core client for 30 sec - exiting 06:59:20 (5368): No heartbeat from core client for 30 sec - exiting 06:59:21 (5368): No heartbeat from core client for 30 sec - exiting 06:59:22 (5368): No heartbeat from core client for 30 sec - exiting 06:59:23 (5368): No heartbeat from core client for 30 sec - exiting 06:59:24 (5368): No heartbeat from core client for 30 sec - exiting 06:59:25 (5368): No heartbeat from core client for 30 sec - exiting 06:59:26 (5368): No heartbeat from core client for 30 sec - exiting 06:59:27 (5368): No heartbeat from core client for 30 sec - exiting 06:59:28 (5368): No heartbeat from core client for 30 sec - exiting 06:59:29 (5368): No heartbeat from core client for 30 sec - exiting 06:59:30 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:53:22 (7728): No heartbeat from core client for 30 sec - exiting 13:53:23 (7728): No heartbeat from core client for 30 sec - exiting 13:53:24 (7728): No heartbeat from core client for 30 sec - exiting 13:53:25 (7728): No heartbeat from core client for 30 sec - exiting 18:58:30 (7728): No heartbeat from core client for 30 sec - exiting 18:58:31 (7728): No heartbeat from core client for 30 sec - exiting 18:58:32 (7728): No heartbeat from core client for 30 sec - exiting 18:58:34 (7728): No heartbeat from core client for 30 sec - exiting 18:58:39 (7728): No heartbeat from core client for 30 sec - exiting 18:58:40 (7728): No heartbeat from core client for 30 sec - exiting 18:58:42 (7728): No heartbeat from core client for 30 sec - exiting 18:58:44 (7728): No heartbeat from core client for 30 sec - exiting 18:58:47 (7728): No heartbeat from core client for 30 sec - exiting 18:58:48 (7728): No heartbeat from core client for 30 sec - exiting 18:58:52 (7728): No heartbeat from core client for 30 sec - exiting 18:58:53 (7728): No heartbeat from core client for 30 sec - exiting 18:58:56 (7728): No heartbeat from core client for 30 sec - exiting 18:58:59 (7728): No heartbeat from core client for 30 sec - exiting 18:59:01 (7728): No heartbeat from core client for 30 sec - exiting 18:59:03 (7728): No heartbeat from core client for 30 sec - exiting 18:59:04 (7728): No heartbeat from core client for 30 sec - exiting 18:59:05 (7728): No heartbeat from core client for 30 sec - exiting 18:59:06 (7728): No heartbeat from core client for 30 sec - exiting 18:59:07 (7728): No heartbeat from core client for 30 sec - exiting 18:59:08 (7728): No heartbeat from core client for 30 sec - exiting 18:59:09 (7728): No heartbeat from core client for 30 sec - exiting 18:59:10 (7728): No heartbeat from core client for 30 sec - exiting 18:59:11 (7728): No heartbeat from core client for 30 sec - exiting 18:59:12 (7728): No heartbeat from core client for 30 sec - exiting 18:59:13 (7728): No heartbeat from core client for 30 sec - exiting 18:59:15 (7728): No heartbeat from core client for 30 sec - exiting 18:59:16 (7728): No heartbeat from core client for 30 sec - exiting 18:59:17 (7728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6784, selfPID=5000, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 7 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1248, selfPID=704, iMonCtr=1 Model crash detected, will try to restart... 17:45:57 (796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4204, selfPID=4204, iMonCtr=2 17:45:59 (796): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8020, selfPID=8020, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:12:04 (5768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:12:38 (3680): No heartbeat from core client for 30 sec - exiting 14:12:39 (3680): No heartbeat from core client for 30 sec - exiting 14:12:40 (3680): No heartbeat from core client for 30 sec - exiting 14:12:41 (3680): No heartbeat from core client for 30 sec - exiting 14:12:42 (3680): No heartbeat from core client for 30 sec - exiting 14:12:43 (3680): No heartbeat from core client for 30 sec - exiting 14:12:44 (3680): No heartbeat from core client for 30 sec - exiting 14:12:45 (3680): No heartbeat from core client for 30 sec - exiting 14:12:46 (3680): No heartbeat from core client for 30 sec - exiting 14:12:47 (3680): No heartbeat from core client for 30 sec - exiting 14:12:48 (3680): No heartbeat from core client for 30 sec - exiting 14:12:49 (3680): No heartbeat from core client for 30 sec - exiting 14:12:50 (3680): No heartbeat from core client for 30 sec - exiting 14:12:51 (3680): No heartbeat from core client for 30 sec - exiting 14:12:52 (3680): No heartbeat from core client for 30 sec - exiting 14:12:53 (3680): No heartbeat from core client for 30 sec - exiting 14:12:54 (3680): No heartbeat from core client for 30 sec - exiting 14:12:55 (3680): No heartbeat from core client for 30 sec - exiting 14:12:56 (3680): No heartbeat from core client for 30 sec - exiting 14:12:58 (3680): No heartbeat from core client for 30 sec - exiting 14:12:59 (3680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:48 (4172): No heartbeat from core client for 30 sec - exiting 20:00:49 (4172): No heartbeat from core client for 30 sec - exiting 20:00:51 (4172): No heartbeat from core client for 30 sec - exiting 20:00:52 (4172): No heartbeat from core client for 30 sec - exiting 20:00:53 (4172): No heartbeat from core client for 30 sec - exiting 20:00:54 (4172): No heartbeat from core client for 30 sec - exiting 20:00:55 (4172): No heartbeat from core client for 30 sec - exiting 20:00:57 (4172): No heartbeat from core client for 30 sec - exiting 20:00:58 (4172): No heartbeat from core client for 30 sec - exiting 20:00:59 (4172): No heartbeat from core client for 30 sec - exiting 20:01:00 (4172): No heartbeat from core client for 30 sec - exiting 20:01:02 (4172): No heartbeat from core client for 30 sec - exiting 20:01:04 (4172): No heartbeat from core client for 30 sec - exiting 20:01:05 (4172): No heartbeat from core client for 30 sec - exiting 20:01:12 (4172): No heartbeat from core client for 30 sec - exiting 20:01:14 (4172): No heartbeat from core client for 30 sec - exiting 20:01:20 (4172): No heartbeat from core client for 30 sec - exiting 20:01:28 (4172): No heartbeat from core client for 30 sec - exiting 20:01:34 (4172): No heartbeat from core client for 30 sec - exiting 20:01:38 (4172): No heartbeat from core client for 30 sec - exiting 20:01:39 (4172): No heartbeat from core client for 30 sec - exiting 20:01:40 (4172): No heartbeat from core client for 30 sec - exiting 20:01:42 (4172): No heartbeat from core client for 30 sec - exiting 20:01:43 (4172): No heartbeat from core client for 30 sec - exiting 20:01:44 (4172): No heartbeat from core client for 30 sec - exiting 20:01:46 (4172): No heartbeat from core client for 30 sec - exiting 20:01:47 (4172): No heartbeat from core client for 30 sec - exiting 20:01:48 (4172): No heartbeat from core client for 30 sec - exiting 20:01:49 (4172): No heartbeat from core client for 30 sec - exiting 20:01:50 (4172): No heartbeat from core client for 30 sec - exiting 20:01:51 (4172): No heartbeat from core client for 30 sec - exiting 20:01:52 (4172): No heartbeat from core client for 30 sec - exiting 20:01:53 (4172): No heartbeat from core client for 30 sec - exiting 20:01:55 (4172): No heartbeat from core client for 30 sec - exiting 20:01:56 (4172): No heartbeat from core client for 30 sec - exiting 20:01:57 (4172): No heartbeat from core client for 30 sec - exiting 20:01:58 (4172): No heartbeat from core client for 30 sec - exiting 20:01:59 (4172): No heartbeat from core client for 30 sec - exiting 20:02:00 (4172): No heartbeat from core client for 30 sec - exiting 20:02:01 (4172): No heartbeat from core client for 30 sec - exiting 20:02:02 (4172): No heartbeat from core client for 30 sec - exiting 20:02:03 (4172): No heartbeat from core client for 30 sec - exiting 20:02:05 (4172): No heartbeat from core client for 30 sec - exiting 20:02:07 (4172): No heartbeat from core client for 30 sec - exiting 20:02:09 (4172): No heartbeat from core client for 30 sec - exiting 20:02:14 (4172): No heartbeat from core client for 30 sec - exiting 20:02:15 (4172): No heartbeat from core client for 30 sec - exiting 20:02:16 (4172): No heartbeat from core client for 30 sec - exiting 20:02:17 (4172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:09:42 (1196): No heartbeat from core client for 30 sec - exiting 07:09:43 (1196): No heartbeat from core client for 30 sec - exiting 07:09:44 (1196): No heartbeat from core client for 30 sec - exiting 07:09:45 (1196): No heartbeat from core client for 30 sec - exiting 07:09:46 (1196): No heartbeat from core client for 30 sec - exiting 07:09:47 (1196): No heartbeat from core client for 30 sec - exiting 07:09:48 (1196): No heartbeat from core client for 30 sec - exiting 07:09:49 (1196): No heartbeat from core client for 30 sec - exiting 07:09:50 (1196): No heartbeat from core client for 30 sec - exiting 07:09:51 (1196): No heartbeat from core client for 30 sec - exiting 07:09:52 (1196): No heartbeat from core client for 30 sec - exiting 07:09:53 (1196): No heartbeat from core client for 30 sec - exiting 07:09:54 (1196): No heartbeat from core client for 30 sec - exiting 07:09:55 (1196): No heartbeat from core client for 30 sec - exiting 07:09:56 (1196): No heartbeat from core client for 30 sec - exiting 07:09:57 (1196): No heartbeat from core client for 30 sec - exiting 07:09:58 (1196): No heartbeat from core client for 30 sec - exiting 07:09:59 (1196): No heartbeat from core client for 30 sec - exiting 07:10:00 (1196): No heartbeat from core client for 30 sec - exiting 07:10:01 (1196): No heartbeat from core client for 30 sec - exiting 07:10:02 (1196): No heartbeat from core client for 30 sec - exiting 07:10:03 (1196): No heartbeat from core client for 30 sec - exiting 07:10:04 (1196): No heartbeat from core client for 30 sec - exiting 07:10:05 (1196): No heartbeat from core client for 30 sec - exiting 07:10:06 (1196): No heartbeat from core client for 30 sec - exiting 07:10:07 (1196): No heartbeat from core client for 30 sec - exiting 07:10:08 (1196): No heartbeat from core client for 30 sec - exiting 07:10:09 (1196): No heartbeat from core client for 30 sec - exiting 07:10:10 (1196): No heartbeat from core client for 30 sec - exiting 07:10:11 (1196): No heartbeat from core client for 30 sec - exiting 07:10:12 (1196): No heartbeat from core client for 30 sec - exiting 07:10:13 (1196): No heartbeat from core client for 30 sec - exiting 07:10:14 (1196): No heartbeat from core client for 30 sec - exiting 07:10:15 (1196): No heartbeat from core client for 30 sec - exiting 07:10:16 (1196): No heartbeat from core client for 30 sec - exiting 07:10:17 (1196): No heartbeat from core client for 30 sec - exiting 07:10:18 (1196): No heartbeat from core client for 30 sec - exiting 07:10:19 (1196): No heartbeat from core client for 30 sec - exiting 07:10:20 (1196): No heartbeat from core client for 30 sec - exiting 07:10:21 (1196): No heartbeat from core client for 30 sec - exiting 07:10:22 (1196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:25:07 (2412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:05:13 (5960): No heartbeat from core client for 30 sec - exiting 07:05:14 (5960): No heartbeat from core client for 30 sec - exiting 07:05:16 (5960): No heartbeat from core client for 30 sec - exiting 07:05:17 (5960): No heartbeat from core client for 30 sec - exiting 07:05:18 (5960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:07:09 (3576): No heartbeat from core client for 30 sec - exiting 07:07:11 (3576): No heartbeat from core client for 30 sec - exiting 07:07:12 (3576): No heartbeat from core client for 30 sec - exiting 07:07:13 (3576): No heartbeat from core client for 30 sec - exiting 07:07:15 (3576): No heartbeat from core client for 30 sec - exiting 07:07:17 (3576): No heartbeat from core client for 30 sec - exiting 07:07:22 (3576): No heartbeat from core client for 30 sec - exiting 07:07:23 (3576): No heartbeat from core client for 30 sec - exiting 07:07:24 (3576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:03:26 (4216): No heartbeat from core client for 30 sec - exiting 23:03:27 (4216): No heartbeat from core client for 30 sec - exiting 23:03:28 (4216): No heartbeat from core client for 30 sec - exiting 23:03:29 (4216): No heartbeat from core client for 30 sec - exiting 23:03:30 (4216): No heartbeat from core client for 30 sec - exiting 23:03:31 (4216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:02:05 (7272): No heartbeat from core client for 30 sec - exiting 10:02:07 (7272): No heartbeat from core client for 30 sec - exiting 10:02:11 (7272): No heartbeat from core client for 30 sec - exiting 10:02:14 (7272): No heartbeat from core client for 30 sec - exiting 10:02:15 (7272): No heartbeat from core client for 30 sec - exiting 10:02:17 (7272): No heartbeat from core client for 30 sec - exiting 10:02:19 (7272): No heartbeat from core client for 30 sec - exiting 10:02:21 (7272): No heartbeat from core client for 30 sec - exiting 10:02:22 (7272): No heartbeat from core client for 30 sec - exiting 10:02:23 (7272): No heartbeat from core client for 30 sec - exiting 10:02:24 (7272): No heartbeat from core client for 30 sec - exiting 10:02:25 (7272): No heartbeat from core client for 30 sec - exiting 10:02:26 (7272): No heartbeat from core client for 30 sec - exiting 10:02:27 (7272): No heartbeat from core client for 30 sec - exiting 10:02:28 (7272): No heartbeat from core client for 30 sec - exiting 10:02:29 (7272): No heartbeat from core client for 30 sec - exiting 10:02:30 (7272): No heartbeat from core client for 30 sec - exiting 10:02:31 (7272): No heartbeat from core client for 30 sec - exiting 10:02:32 (7272): No heartbeat from core client for 30 sec - exiting 10:02:33 (7272): No heartbeat from core client for 30 sec - exiting 10:02:34 (7272): No heartbeat from core client for 30 sec - exiting 10:02:35 (7272): No heartbeat from core client for 30 sec - exiting 10:02:36 (7272): No heartbeat from core client for 30 sec - exiting 10:02:37 (7272): No heartbeat from core client for 30 sec - exiting 10:02:38 (7272): No heartbeat from core client for 30 sec - exiting 10:02:40 (7272): No heartbeat from core client for 30 sec - exiting 10:02:41 (7272): No heartbeat from core client for 30 sec - exiting 10:02:43 (7272): No heartbeat from core client for 30 sec - exiting 10:02:44 (7272): No heartbeat from core client for 30 sec - exiting 10:02:45 (7272): No heartbeat from core client for 30 sec - exiting 10:02:46 (7272): No heartbeat from core client for 30 sec - exiting 10:02:47 (7272): No heartbeat from core client for 30 sec - exiting 10:02:48 (7272): No heartbeat from core client for 30 sec - exiting 10:02:49 (7272): No heartbeat from core client for 30 sec - exiting 10:02:51 (7272): No heartbeat from core client for 30 sec - exiting 10:02:52 (7272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6348, selfPID=6348, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5012, iMonCtr= 2 del crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5072, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5540, iMonCtr=2 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5784, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3292, selfPID=3780, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6212, selfPID=6212, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:23:27 (6452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... G </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Apr 2013 17:48:03	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	126,816	1,978,345	15.6001
25 Mar 2013 19:59:03	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	115,301	1,794,658	15.5650
24 Mar 2013 21:23:29	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	115,299	1,792,407	15.5457
24 Mar 2013 08:22:00	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	115,297	1,790,259	15.5274
23 Mar 2013 22:48:31	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	115,296	1,788,104	15.5088
17 Mar 2013 12:09:43	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	103,780	1,603,375	15.4497
17 Mar 2013 11:04:31	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	103,777	1,601,209	15.4293
16 Mar 2013 23:51:47	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	103,776	1,599,092	15.4091
22 Feb 2013 19:41:37	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	92,257	1,415,750	15.3457
21 Feb 2013 21:33:56	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	92,256	1,413,492	15.3214
02 Feb 2013 21:39:06	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	80,736	1,221,201	15.1259
19 Jan 2013 20:15:36	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	69,216	1,034,750	14.9496
06 Jan 2013 20:47:12	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	57,696	857,587	14.8639
31 Dec 2012 22:34:02	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	46,176	688,929	14.9196
24 Dec 2012 16:26:12	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	34,656	518,055	14.9485
17 Dec 2012 15:25:43	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	23,136	347,196	15.0067
14 Dec 2012 23:31:34	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	11,619	181,246	15.5991
14 Dec 2012 23:31:34	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	11,617	179,375	15.4407
14 Dec 2012 23:31:34	1091233	15455112	hadam3p_pnw_761z_2004_1_007603087_1	11,616	177,561	15.2859