Task 13126223

Name	hadcm3n_ylum_1900_40_007361176_0
Workunit	7558606
Created	6 Jul 2011, 15:17:31 UTC
Sent	7 Jul 2011, 16:02:50 UTC
Report deadline	6 Oct 2011, 23:30:01 UTC
Received	6 Sep 2011, 22:28:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1103150
Run time	23 days 18 hours 13 min 9 sec
CPU time	22 days 0 hours 34 min 59 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	2.65 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:11:40 (10172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:14:36 (7624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10152, iMonCtr=1 Model crash detected, will try to restart... 08:20:57 (7196): No heartbeat from core client for 30 sec - exiting 08:20:58 (7196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 02:38:51 (10844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:49:13 (4164): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 01:26:02 (10372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 02:00:02 (7212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:21:52 (5764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:37:00 (9560): No heartbeat from core client for 30 sec - exiting 01:37:01 (9560): No heartbeat from core client for 30 sec - exiting 01:37:02 (9560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:20:57 (6200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9312, iMonCtr=1 Model crash detected, will try to restart... 10:14:06 (8740): No heartbeat from core client for 30 sec - exiting 10:14:07 (8740): No heartbeat from core client for 30 sec - exiting 10:14:08 (8740): No heartbeat from core client for 30 sec - exiting 10:14:09 (8740): No heartbeat from core client for 30 sec - exiting 10:14:10 (8740): No heartbeat from core client for 30 sec - exiting 10:14:11 (8740): No heartbeat from core client for 30 sec - exiting 10:14:12 (8740): No heartbeat from core client for 30 sec - exiting 10:14:13 (8740): No heartbeat from core client for 30 sec - exiting 10:14:14 (8740): No heartbeat from core client for 30 sec - exiting 10:14:15 (8740): No heartbeat from core client for 30 sec - exiting 10:14:16 (8740): No heartbeat from core client for 30 sec - exiting 10:14:17 (8740): No heartbeat from core client for 30 sec - exiting 10:14:18 (8740): No heartbeat from core client for 30 sec - exiting 10:14:19 (8740): No heartbeat from core client for 30 sec - exiting 10:14:20 (8740): No heartbeat from core client for 30 sec - exiting 10:14:21 (8740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:14:22 (8740): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:57:40 (6592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:05:42 (7372): No heartbeat from core client for 30 sec - exiting 09:05:43 (7372): No heartbeat from core client for 30 sec - exiting 09:05:44 (7372): No heartbeat from core client for 30 sec - exiting 09:05:45 (7372): No heartbeat from core client for 30 sec - exiting 09:05:46 (7372): No heartbeat from core client for 30 sec - exiting 09:05:47 (7372): No heartbeat from core client for 30 sec - exiting 09:05:48 (7372): No heartbeat from core client for 30 sec - exiting 09:05:49 (7372): No heartbeat from core client for 30 sec - exiting 09:05:50 (7372): No heartbeat from core client for 30 sec - exiting 09:05:51 (7372): No heartbeat from core client for 30 sec - exiting 09:05:52 (7372): No heartbeat from core client for 30 sec - exiting 09:05:53 (7372): No heartbeat from core client for 30 sec - exiting 09:05:54 (7372): No heartbeat from core client for 30 sec - exiting 09:05:55 (7372): No heartbeat from core client for 30 sec - exiting 09:05:56 (7372): No heartbeat from core client for 30 sec - exiting 09:05:57 (7372): No heartbeat from core client for 30 sec - exiting 09:05:58 (7372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:53:59 (5852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:28:14 (8960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:28:15 (8960): No heartbeat from core client for 30 sec - exiting 02:28:59 (7992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:34:22 (7136): No heartbeat from core client for 30 sec - exiting 15:34:23 (7136): No heartbeat from core client for 30 sec - exiting 15:34:24 (7136): No heartbeat from core client for 30 sec - exiting 15:34:25 (7136): No heartbeat from core client for 30 sec - exiting 15:34:26 (7136): No heartbeat from core client for 30 sec - exiting 15:34:27 (7136): No heartbeat from core client for 30 sec - exiting 15:34:28 (7136): No heartbeat from core client for 30 sec - exiting 15:34:29 (7136): No heartbeat from core client for 30 sec - exiting 15:34:30 (7136): No heartbeat from core client for 30 sec - exiting 15:34:31 (7136): No heartbeat from core client for 30 sec - exiting 15:34:32 (7136): No heartbeat from core client for 30 sec - exiting 15:34:33 (7136): No heartbeat from core client for 30 sec - exiting 15:34:34 (7136): No heartbeat from core client for 30 sec - exiting 15:34:35 (7136): No heartbeat from core client for 30 sec - exiting 15:34:36 (7136): No heartbeat from core client for 30 sec - exiting 15:34:37 (7136): No heartbeat from core client for 30 sec - exiting 15:34:38 (7136): No heartbeat from core client for 30 sec - exiting 15:34:39 (7136): No heartbeat from core client for 30 sec - exiting 15:34:40 (7136): No heartbeat from core client for 30 sec - exiting 15:34:41 (7136): No heartbeat from core client for 30 sec - exiting 15:34:42 (7136): No heartbeat from core client for 30 sec - exiting 15:34:43 (7136): No heartbeat from core client for 30 sec - exiting 15:34:44 (7136): No heartbeat from core client for 30 sec - exiting 15:34:45 (7136): No heartbeat from core client for 30 sec - exiting 15:34:46 (7136): No heartbeat from core client for 30 sec - exiting 15:34:47 (7136): No heartbeat from core client for 30 sec - exiting 15:34:48 (7136): No heartbeat from core client for 30 sec - exiting 15:34:49 (7136): No heartbeat from core client for 30 sec - exiting 15:34:50 (7136): No heartbeat from core client for 30 sec - exiting 15:34:51 (7136): No heartbeat from core client for 30 sec - exiting 15:34:52 (7136): No heartbeat from core client for 30 sec - exiting 15:34:53 (7136): No heartbeat from core client for 30 sec - exiting 15:34:54 (7136): No heartbeat from core client for 30 sec - exiting 15:34:55 (7136): No heartbeat from core client for 30 sec - exiting 15:34:56 (7136): No heartbeat from core client for 30 sec - exiting 15:34:57 (7136): No heartbeat from core client for 30 sec - exiting 15:34:58 (7136): No heartbeat from core client for 30 sec - exiting 15:34:59 (7136): No heartbeat from core client for 30 sec - exiting 15:35:00 (7136): No heartbeat from core client for 30 sec - exiting 15:35:01 (7136): No heartbeat from core client for 30 sec - exiting 15:35:02 (7136): No heartbeat from core client for 30 sec - exiting 15:35:03 (7136): No heartbeat from core client for 30 sec - exiting 15:35:04 (7136): No heartbeat from core client for 30 sec - exiting 15:35:05 (7136): No heartbeat from core client for 30 sec - exiting 15:35:06 (7136): No heartbeat from core client for 30 sec - exiting 15:35:07 (7136): No heartbeat from core client for 30 sec - exiting 15:35:08 (7136): No heartbeat from core client for 30 sec - exiting 15:35:09 (7136): No heartbeat from core client for 30 sec - exiting 15:35:10 (7136): No heartbeat from core client for 30 sec - exiting 15:35:11 (7136): No heartbeat from core client for 30 sec - exiting 15:35:12 (7136): No heartbeat from core client for 30 sec - exiting 15:35:13 (7136): No heartbeat from core client for 30 sec - exiting 15:35:14 (7136): No heartbeat from core client for 30 sec - exiting 15:35:15 (7136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 02:38:46 (11836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:23:42 (8688): No heartbeat from core client for 30 sec - exiting 20:23:43 (8688): No heartbeat from core client for 30 sec - exiting 20:23:44 (8688): No heartbeat from core client for 30 sec - exiting 20:23:45 (8688): No heartbeat from core client for 30 sec - exiting 20:23:46 (8688): No heartbeat from core client for 30 sec - exiting 20:23:47 (8688): No heartbeat from core client for 30 sec - exiting 20:23:48 (8688): No heartbeat from core client for 30 sec - exiting 20:23:49 (8688): No heartbeat from core client for 30 sec - exiting 20:23:50 (8688): No heartbeat from core client for 30 sec - exiting 20:23:51 (8688): No heartbeat from core client for 30 sec - exiting 20:23:52 (8688): No heartbeat from core client for 30 sec - exiting 20:23:53 (8688): No heartbeat from core client for 30 sec - exiting 20:23:54 (8688): No heartbeat from core client for 30 sec - exiting 20:23:55 (8688): No heartbeat from core client for 30 sec - exiting 20:23:56 (8688): No heartbeat from core client for 30 sec - exiting 20:23:57 (8688): No heartbeat from core client for 30 sec - exiting 20:23:58 (8688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:30:39 (8992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Sep 2011 16:11:39	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	1,010,880	1,925,191	1.9045
06 Sep 2011 00:42:16	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	984,960	1,876,559	1.9052
05 Sep 2011 10:13:40	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	959,040	1,827,207	1.9052
04 Sep 2011 15:59:05	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	933,120	1,777,698	1.9051
01 Sep 2011 20:14:43	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	907,200	1,727,678	1.9044
31 Aug 2011 03:13:18	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	881,280	1,677,982	1.9040
28 Aug 2011 22:35:47	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	855,360	1,628,960	1.9044
26 Aug 2011 04:53:21	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	829,440	1,579,615	1.9044
24 Aug 2011 22:43:20	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	803,520	1,527,788	1.9014
22 Aug 2011 01:24:52	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	777,600	1,475,157	1.8971
18 Aug 2011 09:18:11	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	751,680	1,424,060	1.8945
17 Aug 2011 10:46:50	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	725,760	1,374,357	1.8937
15 Aug 2011 03:36:39	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	699,840	1,326,814	1.8959
13 Aug 2011 04:43:52	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	673,920	1,279,929	1.8992
11 Aug 2011 10:59:08	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	648,000	1,231,543	1.9005
08 Aug 2011 21:54:12	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	622,080	1,181,135	1.8987
07 Aug 2011 04:02:28	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	596,160	1,129,065	1.8939
05 Aug 2011 05:39:11	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	570,240	1,079,532	1.8931
04 Aug 2011 08:30:37	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	544,320	1,028,547	1.8896
03 Aug 2011 12:56:37	1103150	13126223	hadcm3n_ylum_1900_40_007361176_0	518,400	976,072	1.8829