Task 15488193

Name	hadcm3n_3lr1_1940_40_008260280_0
Workunit	8415404
Created	20 Dec 2012, 17:43:14 UTC
Sent	20 Dec 2012, 17:44:00 UTC
Report deadline	22 Mar 2013, 1:11:11 UTC
Received	6 May 2013, 6:12:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	936401
Run time	21 days 21 hours 55 min 49 sec
CPU time	19 days 2 hours 18 min 44 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.66 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 11:26:44 (1076): No heartbeat from core client for 30 sec - exiting 11:26:45 (1076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:26:46 (1076): No heartbeat from core client for 30 sec - exiting 11:26:47 (1076): No heartbeat from core client for 30 sec - exiting 11:26:48 (1076): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3lr1ko.dae1730 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3lr1ko.dae25l0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:00:53 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:07:52 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:22:15 (3972): No heartbeat from core client for 30 sec - exiting 10:22:16 (3972): No heartbeat from core client for 30 sec - exiting 10:22:17 (3972): No heartbeat from core client for 30 sec - exiting 10:22:18 (3972): No heartbeat from core client for 30 sec - exiting 10:22:19 (3972): No heartbeat from core client for 30 sec - exiting 10:22:20 (3972): No heartbeat from core client for 30 sec - exiting 10:22:21 (3972): No heartbeat from core client for 30 sec - exiting 10:22:22 (3972): No heartbeat from core client for 30 sec - exiting 10:22:23 (3972): No heartbeat from core client for 30 sec - exiting 10:22:24 (3972): No heartbeat from core client for 30 sec - exiting 10:22:25 (3972): No heartbeat from core client for 30 sec - exiting 10:22:26 (3972): No heartbeat from core client for 30 sec - exiting 10:22:27 (3972): No heartbeat from core client for 30 sec - exiting 10:22:28 (3972): No heartbeat from core client for 30 sec - exiting 10:22:29 (3972): No heartbeat from core client for 30 sec - exiting 10:22:30 (3972): No heartbeat from core client for 30 sec - exiting 10:22:31 (3972): No heartbeat from core client for 30 sec - exiting 10:22:32 (3972): No heartbeat from core client for 30 sec - exiting 10:22:33 (3972): No heartbeat from core client for 30 sec - exiting 10:22:34 (3972): No heartbeat from core client for 30 sec - exiting 10:22:36 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:22:43 (5728): Can't acquire lockfile (32) - waiting 35s Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3lr1ko.dae4aj0 08:08:22 (4276): No heartbeat from core client for 30 sec - exiting 08:08:23 (4276): No heartbeat from core client for 30 sec - exiting 08:08:24 (4276): No heartbeat from core client for 30 sec - exiting 08:08:25 (4276): No heartbeat from core client for 30 sec - exiting 08:08:26 (4276): No heartbeat from core client for 30 sec - exiting 08:08:27 (4276): No heartbeat from core client for 30 sec - exiting 08:08:28 (4276): No heartbeat from core client for 30 sec - exiting 08:08:29 (4276): No heartbeat from core client for 30 sec - exiting 08:08:30 (4276): No heartbeat from core client for 30 sec - exiting 08:08:31 (4276): No heartbeat from core client for 30 sec - exiting 08:08:32 (4276): No heartbeat from core client for 30 sec - exiting 08:08:33 (4276): No heartbeat from core client for 30 sec - exiting 08:08:34 (4276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:22 (1520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:48:19 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:49:08 (5404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:07:26 (4148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:18 (2116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:18:16 (5460): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:57:14 (4456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:41:34 (4116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:42:35 (3812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3lr1ko.dae61f0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3lr1ko.dae66j0 Ocean Restart file copy failed on 3lr1ko.dae69o0 07:57:55 (5100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:45:19 (2924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 19:32:00 (3600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:20:43 (4420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:23:07 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4084, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 10:49:42 (4052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:30:55 (3904): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:26:46 (4284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:51:52 (5008): No heartbeat from core client for 30 sec - exiting 21:51:53 (5008): No heartbeat from core client for 30 sec - exiting 21:51:54 (5008): No heartbeat from core client for 30 sec - exiting 21:51:55 (5008): No heartbeat from core client for 30 sec - exiting 21:51:56 (5008): No heartbeat from core client for 30 sec - exiting 21:51:57 (5008): No heartbeat from core client for 30 sec - exiting 21:51:58 (5008): No heartbeat from core client for 30 sec - exiting 21:51:59 (5008): No heartbeat from core client for 30 sec - exiting 21:52:00 (5008): No heartbeat from core client for 30 sec - exiting 21:52:01 (5008): No heartbeat from core client for 30 sec - exiting 21:52:02 (5008): No heartbeat from core client for 30 sec - exiting 21:52:03 (5008): No heartbeat from core client for 30 sec - exiting 21:52:04 (5008): No heartbeat from core client for 30 sec - exiting 21:52:05 (5008): No heartbeat from core client for 30 sec - exiting 21:52:06 (5008): No heartbeat from core client for 30 sec - exiting 21:52:08 (5008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3584, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5104, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4380, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4380, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 18:41:42 (4800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:38:22 (4648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:27:17 (2344): No heartbeat from core client for 30 sec - exiting 14:27:19 (2344): No heartbeat from core client for 30 sec - exiting 14:27:20 (2344): No heartbeat from core client for 30 sec - exiting 14:27:21 (2344): No heartbeat from core client for 30 sec - exiting 14:27:23 (2344): No heartbeat from core client for 30 sec - exiting 14:27:24 (2344): No heartbeat from core client for 30 sec - exiting 14:27:25 (2344): No heartbeat from core client for 30 sec - exiting 14:27:26 (2344): No heartbeat from core client for 30 sec - exiting 14:27:27 (2344): No heartbeat from core client for 30 sec - exiting 14:27:28 (2344): No heartbeat from core client for 30 sec - exiting 14:27:29 (2344): No heartbeat from core client for 30 sec - exiting 14:27:30 (2344): No heartbeat from core client for 30 sec - exiting 14:27:31 (2344): No heartbeat from core client for 30 sec - exiting 14:27:32 (2344): No heartbeat from core client for 30 sec - exiting 14:27:33 (2344): No heartbeat from core client for 30 sec - exiting 14:27:34 (2344): No heartbeat from core client for 30 sec - exiting 14:27:35 (2344): No heartbeat from core client for 30 sec - exiting 14:27:36 (2344): No heartbeat from core client for 30 sec - exiting 14:27:37 (2344): No heartbeat from core client for 30 sec - exiting 14:27:38 (2344): No heartbeat from core client for 30 sec - exiting 14:27:39 (2344): No heartbeat from core client for 30 sec - exiting 14:27:40 (2344): No heartbeat from core client for 30 sec - exiting 14:27:41 (2344): No heartbeat from core client for 30 sec - exiting 14:27:42 (2344): No heartbeat from core client for 30 sec - exiting 14:27:43 (2344): No heartbeat from core client for 30 sec - exiting 14:27:44 (2344): No heartbeat from core client for 30 sec - exiting 14:27:45 (2344): No heartbeat from core client for 30 sec - exiting 14:27:46 (2344): No heartbeat from core client for 30 sec - exiting 14:27:47 (2344): No heartbeat from core client for 30 sec - exiting 14:27:48 (2344): No heartbeat from core client for 30 sec - exiting 14:27:49 (2344): No heartbeat from core client for 30 sec - exiting 14:27:50 (2344): No heartbeat from core client for 30 sec - exiting 14:27:51 (2344): No heartbeat from core client for 30 sec - exiting 14:27:52 (2344): No heartbeat from core client for 30 sec - exiting 14:27:53 (2344): No heartbeat from core client for 30 sec - exiting 14:27:54 (2344): No heartbeat from core client for 30 sec - exiting 14:27:55 (2344): No heartbeat from core client for 30 sec - exiting 14:27:56 (2344): No heartbeat from core client for 30 sec - exiting 14:27:57 (2344): No heartbeat from core client for 30 sec - exiting 14:27:58 (2344): No heartbeat from core client for 30 sec - exiting 14:27:59 (2344): No heartbeat from core client for 30 sec - exiting 14:28:00 (2344): No heartbeat from core client for 30 sec - exiting 14:28:01 (2344): No heartbeat from core client for 30 sec - exiting 14:28:02 (2344): No heartbeat from core client for 30 sec - exiting 14:28:03 (2344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:39:54 (3324): No heartbeat from core client for 30 sec - exiting 19:39:55 (3324): No heartbeat from core client for 30 sec - exiting 19:39:56 (3324): No heartbeat from core client for 30 sec - exiting 19:39:57 (3324): No heartbeat from core client for 30 sec - exiting 19:39:58 (3324): No heartbeat from core client for 30 sec - exiting 19:39:59 (3324): No heartbeat from core client for 30 sec - exiting 19:40:00 (3324): No heartbeat from core client for 30 sec - exiting 19:40:01 (3324): No heartbeat from core client for 30 sec - exiting 19:40:02 (3324): No heartbeat from core client for 30 sec - exiting 19:40:03 (3324): No heartbeat from core client for 30 sec - exiting 19:40:04 (3324): No heartbeat from core client for 30 sec - exiting 19:40:05 (3324): No heartbeat from core client for 30 sec - exiting 19:40:06 (3324): No heartbeat from core client for 30 sec - exiting 19:40:07 (3324): No heartbeat from core client for 30 sec - exiting 19:40:08 (3324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:40:52 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:56:01 (4176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2788, iMonCtr=1 Model crash detected, will try to restart... 12:29:50 (584): No heartbeat from core client for 30 sec - exiting 12:29:51 (584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:48:29 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3708, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:47:41 (4384): No heartbeat from core client for 30 sec - exiting 17:47:42 (4384): No heartbeat from core client for 30 sec - exiting 17:47:43 (4384): No heartbeat from core client for 30 sec - exiting 17:47:44 (4384): No heartbeat from core client for 30 sec - exiting 17:47:45 (4384): No heartbeat from core client for 30 sec - exiting 17:47:46 (4384): No heartbeat from core client for 30 sec - exiting 17:47:47 (4384): No heartbeat from core client for 30 sec - exiting 17:47:48 (4384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4144, iMonCtr=1 Model crash detected, will try to restart... 20:52:47 (4508): No heartbeat from core client for 30 sec - exiting 20:52:48 (4508): No heartbeat from core client for 30 sec - exiting 20:52:49 (4508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5736, iMonCtr=1 Model crash detected, will try to restart... 15:58:54 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3680, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3lr1ko.daf46l0 Suspended CPDN Monitor - Suspend request from BOINC... 20:05:13 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5784, iMonCtr=1 Model crash detected, will try to restart... 11:45:37 (4568): No heartbeat from core client for 30 sec - exiting 11:45:38 (4568): No heartbeat from core client for 30 sec - exiting 11:45:39 (4568): No heartbeat from core client for 30 sec - exiting 11:45:40 (4568): No heartbeat from core client for 30 sec - exiting 11:45:41 (4568): No heartbeat from core client for 30 sec - exiting 11:45:42 (4568): No heartbeat from core client for 30 sec - exiting 11:45:43 (4568): No heartbeat from core client for 30 sec - exiting 11:45:44 (4568): No heartbeat from core client for 30 sec - exiting 11:45:45 (4568): No heartbeat from core client for 30 sec - exiting 11:45:46 (4568): No heartbeat from core client for 30 sec - exiting 11:45:47 (4568): No heartbeat from core client for 30 sec - exiting 11:45:48 (4568): No heartbeat from core client for 30 sec - exiting 11:45:49 (4568): No heartbeat from core client for 30 sec - exiting 11:45:50 (4568): No heartbeat from core client for 30 sec - exiting 11:45:51 (4568): No heartbeat from core client for 30 sec - exiting 11:45:52 (4568): No heartbeat from core client for 30 sec - exiting 11:45:53 (4568): No heartbeat from core client for 30 sec - exiting 11:45:54 (4568): No heartbeat from core client for 30 sec - exiting 11:45:55 (4568): No heartbeat from core client for 30 sec - exiting 11:45:56 (4568): No heartbeat from core client for 30 sec - exiting 11:45:57 (4568): No heartbeat from core client for 30 sec - exiting 11:45:58 (4568): No heartbeat from core client for 30 sec - exiting 11:45:59 (4568): No heartbeat from core client for 30 sec - exiting 11:46:00 (4568): No heartbeat from core client for 30 sec - exiting 11:46:01 (4568): No heartbeat from core client for 30 sec - exiting 11:46:02 (4568): No heartbeat from core client for 30 sec - exiting 11:46:03 (4568): No heartbeat from core client for 30 sec - exiting 11:46:04 (4568): No heartbeat from core client for 30 sec - exiting 11:46:05 (4568): No heartbeat from core client for 30 sec - exiting 11:46:06 (4568): No heartbeat from core client for 30 sec - exiting 11:46:07 (4568): No heartbeat from core client for 30 sec - exiting 11:46:08 (4568): No heartbeat from core client for 30 sec - exiting 11:46:09 (4568): No heartbeat from core client for 30 sec - exiting 11:46:10 (4568): No heartbeat from core client for 30 sec - exiting 11:46:11 (4568): No heartbeat from core client for 30 sec - exiting 11:46:12 (4568): No heartbeat from core client for 30 sec - exiting 11:46:13 (4568): No heartbeat from core client for 30 sec - exiting 11:46:14 (4568): No heartbeat from core client for 30 sec - exiting 11:46:15 (4568): No heartbeat from core client for 30 sec - exiting 11:46:16 (4568): No heartbeat from core client for 30 sec - exiting 11:46:17 (4568): No heartbeat from core client for 30 sec - exiting 11:46:18 (4568): No heartbeat from core client for 30 sec - exiting 11:46:19 (4568): No heartbeat from core client for 30 sec - exiting 11:46:20 (4568): No heartbeat from core client for 30 sec - exiting 11:46:21 (4568): No heartbeat from core client for 30 sec - exiting 11:46:22 (4568): No heartbeat from core client for 30 sec - exiting 11:46:23 (4568): No heartbeat from core client for 30 sec - exiting 11:46:24 (4568): No heartbeat from core client for 30 sec - exiting 11:46:25 (4568): No heartbeat from core client for 30 sec - exiting 11:46:26 (4568): No heartbeat from core client for 30 sec - exiting 11:46:27 (4568): No heartbeat from core client for 30 sec - exiting 11:46:28 (4568): No heartbeat from core client for 30 sec - exiting 11:46:29 (4568): No heartbeat from core client for 30 sec - exiting 11:46:30 (4568): No heartbeat from core client for 30 sec - exiting 11:46:31 (4568): No heartbeat from core client for 30 sec - exiting 11:46:32 (4568): No heartbeat from core client for 30 sec - exiting 11:46:33 (4568): No heartbeat from core client for 30 sec - exiting 11:46:34 (4568): No heartbeat from core client for 30 sec - exiting 11:46:35 (4568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6000, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2596, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4956, iMonCtr=1 Model crash detected, will try to restart... 14:51:45 (5516): No heartbeat from core client for 30 sec - exiting 14:51:46 (5516): No heartbeat from core client for 30 sec - exiting 14:51:47 (5516): No heartbeat from core client for 30 sec - exiting 14:51:48 (5516): No heartbeat from core client for 30 sec - exiting 14:51:49 (5516): No heartbeat from core client for 30 sec - exiting 14:51:50 (5516): No heartbeat from core client for 30 sec - exiting 14:51:51 (5516): No heartbeat from core client for 30 sec - exiting 14:51:52 (5516): No heartbeat from core client for 30 sec - exiting 14:51:53 (5516): No heartbeat from core client for 30 sec - exiting 14:51:54 (5516): No heartbeat from core client for 30 sec - exiting 14:51:55 (5516): No heartbeat from core client for 30 sec - exiting 14:51:56 (5516): No heartbeat from core client for 30 sec - exiting 14:51:57 (5516): No heartbeat from core client for 30 sec - exiting 14:51:58 (5516): No heartbeat from core client for 30 sec - exiting 14:51:59 (5516): No heartbeat from core client for 30 sec - exiting 14:52:00 (5516): No heartbeat from core client for 30 sec - exiting 14:52:01 (5516): No heartbeat from core client for 30 sec - exiting 14:52:02 (5516): No heartbeat from core client for 30 sec - exiting 14:52:03 (5516): No heartbeat from core client for 30 sec - exiting 14:52:04 (5516): No heartbeat from core client for 30 sec - exiting 14:52:05 (5516): No heartbeat from core client for 30 sec - exiting 14:52:06 (5516): No heartbeat from core client for 30 sec - exiting 14:52:07 (5516): No heartbeat from core client for 30 sec - exiting 14:52:08 (5516): No heartbeat from core client for 30 sec - exiting 14:52:09 (5516): No heartbeat from core client for 30 sec - exiting 14:52:10 (5516): No heartbeat from core client for 30 sec - exiting 14:52:11 (5516): No heartbeat from core client for 30 sec - exiting 14:52:12 (5516): No heartbeat from core client for 30 sec - exiting 14:52:13 (5516): No heartbeat from core client for 30 sec - exiting 14:52:14 (5516): No heartbeat from core client for 30 sec - exiting 14:52:15 (5516): No heartbeat from core client for 30 sec - exiting 14:52:16 (5516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:55:03 (6020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:44:40 (2460): No heartbeat from core client for 30 sec - exiting 17:44:41 (2460): No heartbeat from core client for 30 sec - exiting 17:44:42 (2460): No heartbeat from core client for 30 sec - exiting 17:44:43 (2460): No heartbeat from core client for 30 sec - exiting 17:44:44 (2460): No heartbeat from core client for 30 sec - exiting 17:44:45 (2460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CSuspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:51:03 (4852): No heartbeat from core client for 30 sec - exiting 14:51:04 (4852): No heartbeat from core client for 30 sec - exiting 14:51:07 (4852): No heartbeat from core client for 30 sec - exiting 14:51:08 (4852): No heartbeat from core client for 30 sec - exiting 14:51:09 (4852): No heartbeat from core client for 30 sec - exiting 14:51:10 (4852): No heartbeat from core client for 30 sec - exiting 14:51:12 (4852): No heartbeat from core client for 30 sec - exiting 14:51:13 (4852): No heartbeat from core client for 30 sec - exiting 14:51:14 (4852): No heartbeat from core client for 30 sec - exiting 14:51:16 (4852): No heartbeat from core client for 30 sec - exiting 14:51:18 (4852): No heartbeat from core client for 30 sec - exiting 14:51:20 (4852): No heartbeat from core client for 30 sec - exiting 14:51:21 (4852): No heartbeat from core client for 30 sec - exiting 14:51:22 (4852): No heartbeat from core client for 30 sec - exiting 14:51:24 (4852): No heartbeat from core client for 30 sec - exiting 14:51:26 (4852): No heartbeat from core client for 30 sec - exiting 14:51:27 (4852): No heartbeat from core client for 30 sec - exiting 14:51:28 (4852): No heartbeat from core client for 30 sec - exiting 14:51:30 (4852): No heartbeat from core client for 30 sec - exiting 14:51:31 (4852): No heartbeat from core client for 30 sec - exiting 14:51:32 (4852): No heartbeat from core client for 30 sec - exiting 14:51:33 (4852): No heartbeat from core client for 30 sec - exiting 14:51:35 (4852): No heartbeat from core client for 30 sec - exiting 14:51:36 (4852): No heartbeat from core client for 30 sec - exiting 14:51:37 (4852): No heartbeat from core client for 30 sec - exiting 14:51:38 (4852): No heartbeat from core client for 30 sec - exiting 14:51:40 (4852): No heartbeat from core client for 30 sec - exiting 14:51:41 (4852): No heartbeat from core client for 30 sec - exiting 14:51:42 (4852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:53:02 (6060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:53:04 (6060): No heartbeat from core client for 30 sec - exiting Co12:18:33 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2336, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:57:08 (12516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:04:26 (5496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:12:29 (5904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Ocean Restart file copy failed on 3lr1ko.daf72s0 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3lr1ko.daf74f0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4432, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:39:57 (4296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3lr1ko.daf9110 19:35:44 (4332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:57:36 (4124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Cont20:08:37 (4808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5168, iMonCtr=1 Model crash detected, will try to restart... 20:30:41 (3984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:28:17 (5740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Ocean Restart file copy failed on 3lr1ko.dag02m0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5176, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 May 2013 09:35:08	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	518,400	1,649,908	3.1827
27 Apr 2013 17:12:30	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	492,480	1,569,326	3.1866
19 Apr 2013 18:12:11	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	466,560	1,488,859	3.1911
14 Apr 2013 08:36:37	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	440,640	1,408,153	3.1957
07 Apr 2013 13:45:06	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	414,720	1,323,016	3.1901
03 Apr 2013 17:26:55	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	388,800	1,235,836	3.1786
26 Mar 2013 10:52:01	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	362,880	1,145,829	3.1576
18 Mar 2013 18:25:52	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	336,960	1,059,343	3.1438
12 Mar 2013 20:55:02	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	311,040	974,175	3.1320
08 Mar 2013 19:56:30	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	285,120	893,703	3.1345
03 Mar 2013 13:25:11	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	259,200	812,850	3.1360
10 Feb 2013 21:17:30	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	233,280	731,338	3.1350
08 Feb 2013 20:06:06	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	207,360	649,722	3.1333
26 Jan 2013 16:25:45	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	181,440	568,104	3.1311
22 Jan 2013 21:20:29	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	155,520	487,071	3.1319
13 Jan 2013 21:39:25	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	129,600	405,851	3.1316
09 Jan 2013 20:14:49	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	103,680	325,659	3.1410
30 Dec 2012 14:55:52	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	77,760	245,071	3.1516
26 Dec 2012 12:34:44	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	51,840	162,709	3.1387
23 Dec 2012 09:38:15	936401	15488193	hadcm3n_3lr1_1940_40_008260280_0	25,920	82,448	3.1809