Task 12749689

Name	hadcm3n_o6v7_1900_40_007204230_1
Workunit	7402510
Created	28 Mar 2011, 14:18:24 UTC
Sent	28 Mar 2011, 23:10:56 UTC
Report deadline	28 Jun 2011, 6:38:07 UTC
Received	20 Jun 2011, 7:29:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1053526
Run time	38 days 4 hours 16 min 52 sec
CPU time	32 days 10 hours 7 min 1 sec
Validate state	Invalid
Credit	11,508.48
Device peak FLOPS	2.32 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.26</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:49:56 (4812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o6v7ka.daa76f0 Atmos Hold Restart file rename failed on atmos_restart.hold 20:03:01 (4180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4620, iMonCtr=1 Model crash detected, will try to restart... 05:59:14 (1568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 05:32:50 (4308): No heartbeat from core client for 30 sec - exiting 05:32:52 (4308): No heartbeat from core client for 30 sec - exiting 05:32:53 (4308): No heartbeat from core client for 30 sec - exiting 05:32:54 (4308): No heartbeat from core client for 30 sec - exiting 05:32:55 (4308): No heartbeat from core client for 30 sec - exiting 05:32:56 (4308): No heartbeat from core client for 30 sec - exiting 05:32:57 (4308): No heartbeat from core client for 30 sec - exiting 05:32:58 (4308): No heartbeat from core client for 30 sec - exiting 05:32:59 (4308): No heartbeat from core client for 30 sec - exiting 05:33:00 (4308): No heartbeat from core client for 30 sec - exiting 05:33:01 (4308): No heartbeat from core client for 30 sec - exiting 05:33:02 (4308): No heartbeat from core client for 30 sec - exiting 05:33:03 (4308): No heartbeat from core client for 30 sec - exiting 05:33:04 (4308): No heartbeat from core client for 30 sec - exiting 05:33:05 (4308): No heartbeat from core client for 30 sec - exiting 05:33:06 (4308): No heartbeat from core client for 30 sec - exiting 05:33:07 (4308): No heartbeat from core client for 30 sec - exiting 05:33:08 (4308): No heartbeat from core client for 30 sec - exiting 05:33:09 (4308): No heartbeat from core client for 30 sec - exiting 05:33:10 (4308): No heartbeat from core client for 30 sec - exiting 05:33:11 (4308): No heartbeat from core client for 30 sec - exiting 05:33:12 (4308): No heartbeat from core client for 30 sec - exiting 05:33:13 (4308): No heartbeat from core client for 30 sec - exiting 05:33:14 (4308): No heartbeat from core client for 30 sec - exiting 05:33:15 (4308): No heartbeat from core client for 30 sec - exiting 05:33:16 (4308): No heartbeat from core client for 30 sec - exiting 05:33:17 (4308): No heartbeat from core client for 30 sec - exiting 05:33:18 (4308): No heartbeat from core client for 30 sec - exiting 05:33:19 (4308): No heartbeat from core client for 30 sec - exiting 05:33:20 (4308): No heartbeat from core client for 30 sec - exiting 05:33:21 (4308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:33:22 (4308): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=432, iMonCtr=1 Model crash detected, will try to restart... 18:27:38 (2852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:12:04 (2228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:12:06 (2228): No heartbeat from core client for 30 sec - exiting 17:12:07 (2228): No heartbeat from core client for 30 sec - exiting 17:12:08 (2228): No heartbeat from core client for 30 sec - exiting 17:12:09 (2228): No heartbeat from core client for 30 sec - exiting 17:12:10 (2228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:35:09 (3700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:35:10 (3700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10148, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14736, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4576, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5756, iMonCtr=1 Model crash detected, will try to restart... 21:48:31 (4284): No heartbeat from core client for 30 sec - exiting 21:48:32 (4284): No heartbeat from core client for 30 sec - exiting 21:48:33 (4284): No heartbeat from core client for 30 sec - exiting 21:48:34 (4284): No heartbeat from core client for 30 sec - exiting 21:48:35 (4284): No heartbeat from core client for 30 sec - exiting 21:48:36 (4284): No heartbeat from core client for 30 sec - exiting 21:48:37 (4284): No heartbeat from core client for 30 sec - exiting 21:48:38 (4284): No heartbeat from core client for 30 sec - exiting 21:48:39 (4284): No heartbeat from core client for 30 sec - exiting 21:48:40 (4284): No heartbeat from core client for 30 sec - exiting 21:48:41 (4284): No heartbeat from core client for 30 sec - exiting 21:48:42 (4284): No heartbeat from core client for 30 sec - exiting 21:48:43 (4284): No heartbeat from core client for 30 sec - exiting 21:48:44 (4284): No heartbeat from core client for 30 sec - exiting 21:48:45 (4284): No heartbeat from core client for 30 sec - exiting 21:48:46 (4284): No heartbeat from core client for 30 sec - exiting 21:48:47 (4284): No heartbeat from core client for 30 sec - exiting 21:48:48 (4284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:53:35 (2620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:54:39 (9208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:54:40 (9208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6156, iMonCtr=1 Model crash detected, will try to restart... 02:54:45 (5836): No heartbeat from core client for 30 sec - exiting 02:54:46 (5836): No heartbeat from core client for 30 sec - exiting 02:54:47 (5836): No heartbeat from core client for 30 sec - exiting 02:54:48 (5836): No heartbeat from core client for 30 sec - exiting 02:54:49 (5836): No heartbeat from core client for 30 sec - exiting 02:54:50 (5836): No heartbeat from core client for 30 sec - exiting 02:54:51 (5836): No heartbeat from core client for 30 sec - exiting 02:54:52 (5836): No heartbeat from core client for 30 sec - exiting 02:54:53 (5836): No heartbeat from core client for 30 sec - exiting 02:54:54 (5836): No heartbeat from core client for 30 sec - exiting 02:54:55 (5836): No heartbeat from core client for 30 sec - exiting 02:54:56 (5836): No heartbeat from core client for 30 sec - exiting 02:54:57 (5836): No heartbeat from core client for 30 sec - exiting 02:54:58 (5836): No heartbeat from core client for 30 sec - exiting 02:54:59 (5836): No heartbeat from core client for 30 sec - exiting 02:55:00 (5836): No heartbeat from core client for 30 sec - exiting 02:55:01 (5836): No heartbeat from core client for 30 sec - exiting 02:55:02 (5836): No heartbeat from core client for 30 sec - exiting 02:55:03 (5836): No heartbeat from core client for 30 sec - exiting 02:55:04 (5836): No heartbeat from core client for 30 sec - exiting 02:55:05 (5836): No heartbeat from core client for 30 sec - exiting 02:55:06 (5836): No heartbeat from core client for 30 sec - exiting 02:55:07 (5836): No heartbeat from core client for 30 sec - exiting 02:55:08 (5836): No heartbeat from core client for 30 sec - exiting 02:55:09 (5836): No heartbeat from core client for 30 sec - exiting 02:55:10 (5836): No heartbeat from core client for 30 sec - exiting 02:55:11 (5836): No heartbeat from core client for 30 sec - exiting 02:55:12 (5836): No heartbeat from core client for 30 sec - exiting 02:55:14 (5836): No heartbeat from core client for 30 sec - exiting 02:55:15 (5836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:08:21 (4604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:08:22 (4604): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:04:32 (4604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:01:23 (4432): No heartbeat from core client for 30 sec - exiting 05:01:24 (4432): No heartbeat from core client for 30 sec - exiting 05:01:25 (4432): No heartbeat from core client for 30 sec - exiting 05:01:26 (4432): No heartbeat from core client for 30 sec - exiting 05:01:27 (4432): No heartbeat from core client for 30 sec - exiting 05:01:28 (4432): No heartbeat from core client for 30 sec - exiting 05:01:29 (4432): No heartbeat from core client for 30 sec - exiting 05:01:30 (4432): No heartbeat from core client for 30 sec - exiting 05:01:31 (4432): No heartbeat from core client for 30 sec - exiting 05:01:32 (4432): No heartbeat from core client for 30 sec - exiting 05:01:33 (4432): No heartbeat from core client for 30 sec - exiting 05:01:34 (4432): No heartbeat from core client for 30 sec - exiting 05:01:36 (4432): No heartbeat from core client for 30 sec - exiting 05:01:37 (4432): No heartbeat from core client for 30 sec - exiting 05:01:38 (4432): No heartbeat from core client for 30 sec - exiting 05:01:39 (4432): No heartbeat from core client for 30 sec - exiting 05:01:40 (4432): No heartbeat from core client for 30 sec - exiting 05:01:41 (4432): No heartbeat from core client for 30 sec - exiting 05:01:42 (4432): No heartbeat from core client for 30 sec - exiting 05:01:43 (4432): No heartbeat from core client for 30 sec - exiting 05:01:44 (4432): No heartbeat from core client for 30 sec - exiting 05:01:45 (4432): No heartbeat from core client for 30 sec - exiting 05:01:46 (4432): No heartbeat from core client for 30 sec - exiting 05:01:47 (4432): No heartbeat from core client for 30 sec - exiting 05:01:48 (4432): No heartbeat from core client for 30 sec - exiting 05:01:49 (4432): No heartbeat from core client for 30 sec - exiting 05:01:50 (4432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:39 (6380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:25:38 (3552): No heartbeat from core client for 30 sec - exiting 20:25:39 (3552): No heartbeat from core client for 30 sec - exiting 20:25:40 (3552): No heartbeat from core client for 30 sec - exiting 20:25:41 (3552): No heartbeat from core client for 30 sec - exiting 20:25:42 (3552): No heartbeat from core client for 30 sec - exiting 20:25:43 (3552): No heartbeat from core client for 30 sec - exiting 20:25:44 (3552): No heartbeat from core client for 30 sec - exiting 20:25:45 (3552): No heartbeat from core client for 30 sec - exiting 20:25:46 (3552): No heartbeat from core client for 30 sec - exiting 20:25:47 (3552): No heartbeat from core client for 30 sec - exiting 20:25:48 (3552): No heartbeat from core client for 30 sec - exiting 20:25:49 (3552): No heartbeat from core client for 30 sec - exiting 20:25:50 (3552): No heartbeat from core client for 30 sec - exiting 20:25:51 (3552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:25:52 (3552): No heartbeat from core client for 30 sec - exiting 20:25:53 (3552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:55:22 (3484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:55:24 (3484): No heartbeat from core client for 30 sec - exiting 18:55:25 (3484): No heartbeat from core client for 30 sec - exiting 18:55:26 (3484): No heartbeat from core client for 30 sec - exiting 18:55:27 (3484): No heartbeat from core client for 30 sec - exiting 18:55:28 (3484): No heartbeat from core client for 30 sec - exiting 18:55:29 (3484): No heartbeat from core client for 30 sec - exiting 18:55:30 (3484): No heartbeat from core client for 30 sec - exiting 18:55:31 (3484): No heartbeat from core client for 30 sec - exiting 18:55:32 (3484): No heartbeat from core client for 30 sec - exiting 18:55:33 (3484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:20:29 (34008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:01:05 (18220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:54:38 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Jun 2011 15:27:41	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	959,040	2,742,927	2.8601
14 Jun 2011 02:59:45	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	933,120	2,670,961	2.8624
09 Jun 2011 23:08:00	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	907,200	2,597,012	2.8627
06 Jun 2011 09:10:22	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	881,280	2,523,721	2.8637
02 Jun 2011 18:17:19	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	855,360	2,449,544	2.8638
30 May 2011 17:07:46	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	829,440	2,374,041	2.8622
28 May 2011 12:00:34	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	803,520	2,286,705	2.8459
23 May 2011 07:56:52	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	777,600	2,215,892	2.8497
19 May 2011 17:13:32	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	751,680	2,139,917	2.8468
16 May 2011 00:58:18	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	725,760	2,068,650	2.8503
12 May 2011 14:32:04	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	699,840	1,992,899	2.8476
09 May 2011 02:42:11	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	673,920	1,918,804	2.8472
06 May 2011 04:53:32	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	648,000	1,843,218	2.8445
05 May 2011 05:39:11	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	622,080	1,770,322	2.8458
04 May 2011 05:01:01	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	596,160	1,695,850	2.8446
03 May 2011 03:14:43	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	570,240	1,617,967	2.8373
02 May 2011 00:40:35	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	544,320	1,541,203	2.8314
30 Apr 2011 09:14:50	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	518,400	1,463,670	2.8234
29 Apr 2011 07:51:56	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	492,480	1,387,684	2.8177
28 Apr 2011 07:18:36	1053526	12749689	hadcm3n_o6v7_1900_40_007204230_1	466,560	1,311,679	2.8114