Task 15421079

Name	hadcm3n_y8cs_1980_40_008243043_2
Workunit	8398167
Created	30 Oct 2012, 20:16:20 UTC
Sent	30 Oct 2012, 20:16:41 UTC
Report deadline	30 Jan 2013, 3:43:52 UTC
Received	17 Dec 2012, 14:19:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1143518
Run time	9 days 15 hours 50 min 34 sec
CPU time	6 days 11 hours 25 min 25 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	1.79 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5176, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4240, iMonCtr=1 Model crash detected, will try to restart... 08:30:59 (2392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:00 (2392): No heartbeat from core client for 30 sec - exiting 08:35:30 (4888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:14:59 (4252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:15:00 (4252): No heartbeat from core client for 30 sec - exiting 08:15:01 (4252): No heartbeat from core client for 30 sec - exiting 08:15:02 (4252): No heartbeat from core client for 30 sec - exiting 08:15:03 (4252): No heartbeat from core client for 30 sec - exiting 08:19:34 (5204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:19:35 (5204): No heartbeat from core client for 30 sec - exiting 08:29:43 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:29:44 (5052): No heartbeat from core client for 30 sec - exiting 08:29:45 (5052): No heartbeat from core client for 30 sec - exiting 08:29:46 (5052): No heartbeat from core client for 30 sec - exiting 08:35:08 (3644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:09 (3644): No heartbeat from core client for 30 sec - exiting 08:35:10 (3644): No heartbeat from core client for 30 sec - exiting 08:55:44 (3984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:45 (3984): No heartbeat from core client for 30 sec - exiting 08:55:46 (3984): No heartbeat from core client for 30 sec - exiting 09:00:56 (5528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:57 (5528): No heartbeat from core client for 30 sec - exiting 09:00:58 (5528): No heartbeat from core client for 30 sec - exiting 09:00:59 (5528): No heartbeat from core client for 30 sec - exiting 09:01:00 (5528): No heartbeat from core client for 30 sec - exiting 09:06:12 (2724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:06:13 (2724): No heartbeat from core client for 30 sec - exiting 09:06:14 (2724): No heartbeat from core client for 30 sec - exiting 09:10:48 (704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:10:49 (704): No heartbeat from core client for 30 sec - exiting 09:10:50 (704): No heartbeat from core client for 30 sec - exiting 09:10:51 (704): No heartbeat from core client for 30 sec - exiting 09:36:32 (2848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:36:33 (2848): No heartbeat from core client for 30 sec - exiting 09:36:34 (2848): No heartbeat from core client for 30 sec - exiting 09:41:28 (988): No heartbeat from core client for 30 sec - exiting 09:41:29 (988): No heartbeat from core client for 30 sec - exiting 09:41:30 (988): No heartbeat from core client for 30 sec - exiting 09:41:31 (988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:37 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:38 (5800): No heartbeat from core client for 30 sec - exiting 09:56:36 (6820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:01:22 (6264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:01:23 (6264): No heartbeat from core client for 30 sec - exiting 10:01:24 (6264): No heartbeat from core client for 30 sec - exiting 10:11:47 (300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:11:48 (300): No heartbeat from core client for 30 sec - exiting 10:11:49 (300): No heartbeat from core client for 30 sec - exiting 10:11:50 (300): No heartbeat from core client for 30 sec - exiting 10:24:15 (356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:24:16 (356): No heartbeat from core client for 30 sec - exiting 10:24:17 (356): No heartbeat from core client for 30 sec - exiting 10:24:18 (356): No heartbeat from core client for 30 sec - exiting 10:24:19 (356): No heartbeat from core client for 30 sec - exiting 10:27:56 (5776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:57 (5776): No heartbeat from core client for 30 sec - exiting 10:27:58 (5776): No heartbeat from core client for 30 sec - exiting 10:34:30 (2996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:34:31 (2996): No heartbeat from core client for 30 sec - exiting 10:34:32 (2996): No heartbeat from core client for 30 sec - exiting 10:34:33 (2996): No heartbeat from core client for 30 sec - exiting 10:37:54 (6220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:37:55 (6220): No heartbeat from core client for 30 sec - exiting 10:37:56 (6220): No heartbeat from core client for 30 sec - exiting 10:37:57 (6220): No heartbeat from core client for 30 sec - exiting 10:44:31 (5760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:44:32 (5760): No heartbeat from core client for 30 sec - exiting 10:44:33 (5760): No heartbeat from core client for 30 sec - exiting 10:44:34 (5760): No heartbeat from core client for 30 sec - exiting 10:54:58 (5528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:54:59 (5528): No heartbeat from core client for 30 sec - exiting 10:55:00 (5528): No heartbeat from core client for 30 sec - exiting 11:36:47 (6400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:48 (6400): No heartbeat from core client for 30 sec - exiting 11:36:49 (6400): No heartbeat from core client for 30 sec - exiting 11:36:50 (6400): No heartbeat from core client for 30 sec - exiting 11:36:51 (6400): No heartbeat from core client for 30 sec - exiting 11:52:56 (4252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:52:57 (4252): No heartbeat from core client for 30 sec - exiting 12:02:34 (6572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:35 (6572): No heartbeat from core client for 30 sec - exiting 12:02:36 (6572): No heartbeat from core client for 30 sec - exiting 12:02:37 (6572): No heartbeat from core client for 30 sec - exiting 12:02:38 (6572): No heartbeat from core client for 30 sec - exiting 12:06:45 (3772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:06:46 (3772): No heartbeat from core client for 30 sec - exiting 12:06:47 (3772): No heartbeat from core client for 30 sec - exiting 12:06:48 (3772): No heartbeat from core client for 30 sec - exiting 12:06:49 (3772): No heartbeat from core client for 30 sec - exiting 12:12:45 (2500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:18:28 (6788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:18:29 (6788): No heartbeat from core client for 30 sec - exiting 12:18:30 (6788): No heartbeat from core client for 30 sec - exiting 12:23:35 (3064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:23:36 (3064): No heartbeat from core client for 30 sec - exiting 12:23:37 (3064): No heartbeat from core client for 30 sec - exiting 12:23:38 (3064): No heartbeat from core client for 30 sec - exiting 12:23:39 (3064): No heartbeat from core client for 30 sec - exiting 12:48:58 (7028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:48:59 (7028): No heartbeat from core client for 30 sec - exiting 12:53:50 (2232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:53:51 (2232): No heartbeat from core client for 30 sec - exiting 12:53:52 (2232): No heartbeat from core client for 30 sec - exiting 13:02:30 (2492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:02:31 (2492): No heartbeat from core client for 30 sec - exiting 13:08:40 (4836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:08:41 (4836): No heartbeat from core client for 30 sec - exiting 13:08:42 (4836): No heartbeat from core client for 30 sec - exiting 13:08:43 (4836): No heartbeat from core client for 30 sec - exiting 13:24:16 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:17 (5052): No heartbeat from core client for 30 sec - exiting 13:24:18 (5052): No heartbeat from core client for 30 sec - exiting 13:28:59 (1084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:34:49 (6412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:34:50 (6412): No heartbeat from core client for 30 sec - exiting 13:34:51 (6412): No heartbeat from core client for 30 sec - exiting 13:34:52 (6412): No heartbeat from core client for 30 sec - exiting 13:39:55 (2540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:39:56 (2540): No heartbeat from core client for 30 sec - exiting 13:39:57 (2540): No heartbeat from core client for 30 sec - exiting 13:39:58 (2540): No heartbeat from core client for 30 sec - exiting 13:49:45 (6496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:49:46 (6496): No heartbeat from core client for 30 sec - exiting 14:00:04 (3336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:00:05 (3336): No heartbeat from core client for 30 sec - exiting 14:05:15 (2148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:05:16 (2148): No heartbeat from core client for 30 sec - exiting 14:10:09 (5608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:10:10 (5608): No heartbeat from core client for 30 sec - exiting 14:10:11 (5608): No heartbeat from core client for 30 sec - exiting 14:10:12 (5608): No heartbeat from core client for 30 sec - exiting 14:10:13 (5608): No heartbeat from core client for 30 sec - exiting 14:15:25 (6928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:15:26 (6928): No heartbeat from core client for 30 sec - exiting 14:15:27 (6928): No heartbeat from core client for 30 sec - exiting 14:20:29 (5084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:20:30 (5084): No heartbeat from core client for 30 sec - exiting 14:20:31 (5084): No heartbeat from core client for 30 sec - exiting 14:20:32 (5084): No heartbeat from core client for 30 sec - exiting 14:30:17 (2008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:30:18 (2008): No heartbeat from core client for 30 sec - exiting 14:30:19 (2008): No heartbeat from core client for 30 sec - exiting 14:30:20 (2008): No heartbeat from core client for 30 sec - exiting 14:35:48 (3772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:35:49 (3772): No heartbeat from core client for 30 sec - exiting 14:35:50 (3772): No heartbeat from core client for 30 sec - exiting 15:00:30 (4692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:00:31 (4692): No heartbeat from core client for 30 sec - exiting 15:00:32 (4692): No heartbeat from core client for 30 sec - exiting 15:00:33 (4692): No heartbeat from core client for 30 sec - exiting 15:16:04 (100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:21:09 (352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:21:10 (352): No heartbeat from core client for 30 sec - exiting 15:21:11 (352): No heartbeat from core client for 30 sec - exiting 15:31:05 (6992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:31:06 (6992): No heartbeat from core client for 30 sec - exiting 15:31:07 (6992): No heartbeat from core client for 30 sec - exiting 15:31:08 (6992): No heartbeat from core client for 30 sec - exiting 15:36:13 (4908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:36:14 (4908): No heartbeat from core client for 30 sec - exiting 15:36:15 (4908): No heartbeat from core client for 30 sec - exiting 15:36:16 (4908): No heartbeat from core client for 30 sec - exiting 15:36:17 (4908): No heartbeat from core client for 30 sec - exiting 15:41:13 (5960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:41:14 (5960): No heartbeat from core client for 30 sec - exiting 15:51:18 (6948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:51:19 (6948): No heartbeat from core client for 30 sec - exiting 15:51:20 (6948): No heartbeat from core client for 30 sec - exiting 15:56:21 (3568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:56:22 (3568): No heartbeat from core client for 30 sec - exiting 15:56:23 (3568): No heartbeat from core client for 30 sec - exiting 15:56:24 (3568): No heartbeat from core client for 30 sec - exiting 16:01:24 (6992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:01:25 (6992): No heartbeat from core client for 30 sec - exiting 16:11:49 (6364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:11:50 (6364): No heartbeat from core client for 30 sec - exiting 16:16:47 (884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:16:48 (884): No heartbeat from core client for 30 sec - exiting 16:21:47 (2580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:41:59 (5884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:42:00 (5884): No heartbeat from core client for 30 sec - exiting 16:47:01 (7040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:52:03 (5560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:02:01 (4756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:02:02 (4756): No heartbeat from core client for 30 sec - exiting 17:02:03 (4756): No heartbeat from core client for 30 sec - exiting 17:07:19 (6436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:17:33 (988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:17:34 (988): No heartbeat from core client for 30 sec - exiting 17:17:35 (988): No heartbeat from core client for 30 sec - exiting 17:17:36 (988): No heartbeat from core client for 30 sec - exiting 17:17:37 (988): No heartbeat from core client for 30 sec - exiting 17:32:10 (4260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:32:11 (4260): No heartbeat from core client for 30 sec - exiting 17:42:34 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:42:35 (5428): No heartbeat from core client for 30 sec - exiting 17:42:36 (5428): No heartbeat from core client for 30 sec - exiting 17:42:37 (5428): No heartbeat from core client for 30 sec - exiting 17:42:38 (5428): No heartbeat from core client for 30 sec - exiting 17:52:51 (7092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:52:52 (7092): No heartbeat from core client for 30 sec - exiting 17:52:53 (7092): No heartbeat from core client for 30 sec - exiting 17:52:54 (7092): No heartbeat from core client for 30 sec - exiting 18:12:56 (1360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:12:57 (1360): No heartbeat from core client for 30 sec - exiting 18:12:58 (1360): No heartbeat from core client for 30 sec - exiting 18:12:59 (1360): No heartbeat from core client for 30 sec - exiting 18:17:52 (1832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:17:53 (1832): No heartbeat from core client for 30 sec - exiting 18:17:54 (1832): No heartbeat from core client for 30 sec - exiting 18:17:55 (1832): No heartbeat from core client for 30 sec - exiting 18:28:32 (3080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:47 (3892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:48 (3892): No heartbeat from core client for 30 sec - exiting 18:38:49 (3892): No heartbeat from core client for 30 sec - exiting 18:38:50 (3892): No heartbeat from core client for 30 sec - exiting 18:43:33 (2180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:34 (2180): No heartbeat from core client for 30 sec - exiting 18:43:35 (2180): No heartbeat from core client for 30 sec - exiting 18:43:36 (2180): No heartbeat from core client for 30 sec - exiting 18:43:37 (2180): No heartbeat from core client for 30 sec - exiting 18:53:34 (3768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:53:35 (3768): No heartbeat from core client for 30 sec - exiting 18:53:36 (3768): No heartbeat from core client for 30 sec - exiting 18:53:37 (3768): No heartbeat from core client for 30 sec - exiting 19:03:34 (4888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:43 (6264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:44 (6264): No heartbeat from core client for 30 sec - exiting 19:14:23 (1456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:14:24 (1456): No heartbeat from core client for 30 sec - exiting 20:03:51 (996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:05:36 (1196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:05:37 (1196): No heartbeat from core client for 30 sec - exiting 22:05:38 (1196): No heartbeat from core client for 30 sec - exiting 23:10:10 (1956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:10:11 (1956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 00:53:15 (5680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:53:16 (5680): No heartbeat from core client for 30 sec - exiting 00:53:17 (5680): No heartbeat from core client for 30 sec - exiting 00:53:18 (5680): No heartbeat from core client for 30 sec - exiting 00:53:19 (5680): No heartbeat from core client for 30 sec - exiting 00:53:20 (5680): No heartbeat from core client for 30 sec - exiting 00:56:07 (4132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:08:36 (2724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:08:37 (2724): No heartbeat from core client for 30 sec - exiting 01:50:26 (2668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:50:27 (2668): No heartbeat from core client for 30 sec - exiting 01:50:28 (2668): No heartbeat from core client for 30 sec - exiting 01:55:03 (2940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:06:34 (952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:06:35 (952): No heartbeat from core client for 30 sec - exiting 02:06:36 (952): No heartbeat from core client for 30 sec - exiting 05:15:09 (2532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:29 (1576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:30 (1576): No heartbeat from core client for 30 sec - exiting 05:20:31 (1576): No heartbeat from core client for 30 sec - exiting 05:20:32 (1576): No heartbeat from core client for 30 sec - exiting 06:11:38 (7164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:11:39 (7164): No heartbeat from core client for 30 sec - exiting 06:11:40 (7164): No heartbeat from core client for 30 sec - exiting 06:11:41 (7164): No heartbeat from core client for 30 sec - exiting 08:03:35 (6172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:08:21 (4812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:22:59 (4268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:00 (4268): No heartbeat from core client for 30 sec - exiting 08:23:01 (4268): No heartbeat from core client for 30 sec - exiting 08:29:21 (4600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:29:22 (4600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:01:35 (6316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:01:36 (6316): No heartbeat from core client for 30 sec - exiting 09:01:37 (6316): No heartbeat from core client for 30 sec - exiting 09:01:38 (6316): No heartbeat from core client for 30 sec - exiting 09:01:39 (6316): No heartbeat from core client for 30 sec - exiting 09:01:40 (6316): No heartbeat from core client for 30 sec - exiting 09:06:11 (6824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:06:12 (6824): No heartbeat from core client for 30 sec - exiting 09:06:13 (6824): No heartbeat from core client for 30 sec - exiting 09:06:14 (6824): No heartbeat from core client for 30 sec - exiting 09:40:02 (7116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:40:03 (7116): No heartbeat from core client for 30 sec - exiting 09:40:04 (7116): No heartbeat from core client for 30 sec - exiting 09:40:05 (7116): No heartbeat from core client for 30 sec - exiting 09:40:06 (7116): No heartbeat from core client for 30 sec - exiting 09:51:56 (5324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:57 (5324): No heartbeat from core client for 30 sec - exiting 09:51:58 (5324): No heartbeat from core client for 30 sec - exiting 09:57:38 (2088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:39 (2088): No heartbeat from core client for 30 sec - exiting 09:57:40 (2088): No heartbeat from core client for 30 sec - exiting 09:57:41 (2088): No heartbeat from core client for 30 sec - exiting 10:02:40 (6224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:02:41 (6224): No heartbeat from core client for 30 sec - exiting 10:02:42 (6224): No heartbeat from core client for 30 sec - exiting 10:07:53 (5600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:54 (5600): No heartbeat from core client for 30 sec - exiting 10:07:55 (5600): No heartbeat from core client for 30 sec - exiting 10:07:56 (5600): No heartbeat from core client for 30 sec - exiting 10:12:56 (3276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:22:56 (2112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:28:10 (328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:38:01 (6804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:38:02 (6804): No heartbeat from core client for 30 sec - exiting 10:38:03 (6804): No heartbeat from core client for 30 sec - exiting 10:38:04 (6804): No heartbeat from core client for 30 sec - exiting 10:38:05 (6804): No heartbeat from core client for 30 sec - exiting 10:43:13 (6544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:08:34 (3840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:08:35 (3840): No heartbeat from core client for 30 sec - exiting 11:08:36 (3840): No heartbeat from core client for 30 sec - exiting 11:09:44 (2112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:04:52 (3336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:06:02 (2692): No heartbeat from core client for 30 sec - exiting 12:06:03 (2692): No heartbeat from core client for 30 sec - exiting 12:06:04 (2692): No heartbeat from core client for 30 sec - exiting 12:06:05 (2692): No heartbeat from core client for 30 sec - exiting 12:06:06 (2692): No heartbeat from core client for 30 sec - exiting 12:06:07 (2692): No heartbeat from core client for 30 sec - exiting 12:06:08 (2692): No heartbeat from core client for 30 sec - exiting 12:06:09 (2692): No heartbeat from core client for 30 sec - exiting 12:06:10 (2692): No heartbeat from core client for 30 sec - exiting 12:06:11 (2692): No heartbeat from core client for 30 sec - exiting 12:06:12 (2692): No heartbeat from core client for 30 sec - exiting 12:06:13 (2692): No heartbeat from core client for 30 sec - exiting 12:06:14 (2692): No heartbeat from core client for 30 sec - exiting 12:06:15 (2692): No heartbeat from core client for 30 sec - exiting 12:06:16 (2692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:12:43 (6496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:12:44 (6496): No heartbeat from core client for 30 sec - exiting 14:12:45 (6496): No heartbeat from core client for 30 sec - exiting 14:28:01 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:28:53 (6244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:28:54 (6244): No heartbeat from core client for 30 sec - exiting 15:28:55 (6244): No heartbeat from core client for 30 sec - exiting 15:48:03 (3028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:09:37 (2520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:09:38 (2520): No heartbeat from core client for 30 sec - exiting 16:09:39 (2520): No heartbeat from core client for 30 sec - exiting 16:09:40 (2520): No heartbeat from core client for 30 sec - exiting 16:09:41 (2520): No heartbeat from core client for 30 sec - exiting 16:27:35 (4476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:27:36 (4476): No heartbeat from core client for 30 sec - exiting 16:27:37 (4476): No heartbeat from core client for 30 sec - exiting 16:27:38 (4476): No heartbeat from core client for 30 sec - exiting 16:53:39 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:53:40 (5772): No heartbeat from core client for 30 sec - exiting 16:53:41 (5772): No heartbeat from core client for 30 sec - exiting 16:55:14 (5124): No heartbeat from core client for 30 sec - exiting 16:55:15 (5124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:16 (5124): No heartbeat from core client for 30 sec - exiting 16:55:17 (5124): No heartbeat from core client for 30 sec - exiting 16:55:18 (5124): No heartbeat from core client for 30 sec - exiting 18:03:52 (1472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:03:53 (1472): No heartbeat from core client for 30 sec - exiting 18:03:54 (1472): No heartbeat from core client for 30 sec - exiting 20:02:58 (4004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:22 (3892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:23 (3892): No heartbeat from core client for 30 sec - exiting 22:27:24 (3892): No heartbeat from core client for 30 sec - exiting 23:03:52 (2512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:54:26 (2460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:54:27 (2460): No heartbeat from core client for 30 sec - exiting 00:54:28 (2460): No heartbeat from core client for 30 sec - exiting 01:02:18 (6484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:02:19 (6484): No heartbeat from core client for 30 sec - exiting 01:02:20 (6484): No heartbeat from core client for 30 sec - exiting 01:02:21 (6484): No heartbeat from core client for 30 sec - exiting 01:02:22 (6484): No heartbeat from core client for 30 sec - exiting 01:12:01 (4600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:12:02 (4600): No heartbeat from core client for 30 sec - exiting 01:12:03 (4600): No heartbeat from core client for 30 sec - exiting 01:12:04 (4600): No heartbeat from core client for 30 sec - exiting 01:12:05 (4600): No heartbeat from core client for 30 sec - exiting 02:05:50 (1132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:05:51 (1132): No heartbeat from core client for 30 sec - exiting 02:05:52 (1132): No heartbeat from core client for 30 sec - exiting 02:05:53 (1132): No heartbeat from core client for 30 sec - exiting 02:07:31 (2612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:07:32 (2612): No heartbeat from core client for 30 sec - exiting 02:07:33 (2612): No heartbeat from core client for 30 sec - exiting 02:07:34 (2612): No heartbeat from core client for 30 sec - exiting 02:07:35 (2612): No heartbeat from core client for 30 sec - exiting 02:07:36 (2612): No heartbeat from core client for 30 sec - exiting 03:07:02 (4992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:07:03 (4992): No heartbeat from core client for 30 sec - exiting 03:09:01 (3500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:09:02 (3500): No heartbeat from core client for 30 sec - exiting 03:09:03 (3500): No heartbeat from core client for 30 sec - exiting 03:09:04 (3500): No heartbeat from core client for 30 sec - exiting 03:09:05 (3500): No heartbeat from core client for 30 sec - exiting 03:09:06 (3500): No heartbeat from core client for 30 sec - exiting 05:09:12 (6472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:10:17 (2696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:10:18 (2696): No heartbeat from core client for 30 sec - exiting 06:10:19 (2696): No heartbeat from core client for 30 sec - exiting 07:17:11 (5668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:13 (5668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5952, iMonCtr=1 Model crash detected, will try to restart... 02:59:57 (996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:59:58 (996): No heartbeat from core client for 30 sec - exiting 02:59:59 (996): No heartbeat from core client for 30 sec - exiting 03:00:00 (996): No heartbeat from core client for 30 sec - exiting 03:00:09 (5680): Can't acquire lockfile (32) - waiting 35s 03:13:24 (5680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:54:32 (520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=984, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4876, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4876, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4876, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4876, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3596, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Nov 2012 14:54:49	1143518	15421079	hadcm3n_y8cs_1980_40_008243043_2	207,360	498,772	2.4053
08 Nov 2012 15:45:37	1143518	15421079	hadcm3n_y8cs_1980_40_008243043_2	181,440	436,814	2.4075
07 Nov 2012 16:38:38	1143518	15421079	hadcm3n_y8cs_1980_40_008243043_2	155,520	374,529	2.4082
06 Nov 2012 15:11:32	1143518	15421079	hadcm3n_y8cs_1980_40_008243043_2	129,600	312,357	2.4102
04 Nov 2012 02:04:41	1143518	15421079	hadcm3n_y8cs_1980_40_008243043_2	103,680	249,275	2.4043
03 Nov 2012 03:49:07	1143518	15421079	hadcm3n_y8cs_1980_40_008243043_2	77,760	187,616	2.4128
02 Nov 2012 02:49:08	1143518	15421079	hadcm3n_y8cs_1980_40_008243043_2	51,840	124,810	2.4076
01 Nov 2012 00:55:12	1143518	15421079	hadcm3n_y8cs_1980_40_008243043_2	25,920	61,910	2.3885