Task 15728291

Name	hadcm3n_zcz8_1960_40_008349795_0
Workunit	8500656
Created	17 Apr 2013, 16:55:30 UTC
Sent	17 Apr 2013, 16:55:52 UTC
Report deadline	18 Jul 2013, 0:23:03 UTC
Received	26 Apr 2013, 12:43:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1239995
Run time	3 days 22 hours 27 min 34 sec
CPU time	3 days 8 hours 51 min 26 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	4.22 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... OPEN: Unable to Open File dataout/zcz8ka.pdg4c10 for Read/Write Model crashed: STWORK : Error opening output PP file on unit 63 tmp/pipe_dummy 2048 OPEN: Unable to Open File dataout/zcz8ka.pdg4c10 for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/pipe_dummy 2048 OPEN: Unable to Open File dataout/zcz8ka.pdg4c10 for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/pipe_dummy 2048 Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:40:40 (1032): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 08:18:11 (91744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:18:12 (91744): No heartbeat from core client for 30 sec - exiting 08:18:13 (91744): No heartbeat from core client for 30 sec - exiting 08:18:14 (91744): No heartbeat from core client for 30 sec - exiting 08:19:06 (87220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:19:07 (87220): No heartbeat from core client for 30 sec - exiting 08:19:08 (87220): No heartbeat from core client for 30 sec - exiting 08:19:09 (87220): No heartbeat from core client for 30 sec - exiting 08:19:10 (87220): No heartbeat from core client for 30 sec - exiting 08:19:11 (87220): No heartbeat from core client for 30 sec - exiting 08:19:12 (87220): No heartbeat from core client for 30 sec - exiting 08:19:13 (87220): No heartbeat from core client for 30 sec - exiting 08:19:14 (87220): No heartbeat from core client for 30 sec - exiting 08:19:15 (87220): No heartbeat from core client for 30 sec - exiting 08:25:34 (95496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:27:44 (25560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:52 (98200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:53 (98200): No heartbeat from core client for 30 sec - exiting 08:31:54 (98200): No heartbeat from core client for 30 sec - exiting 08:31:55 (98200): No heartbeat from core client for 30 sec - exiting 08:31:56 (98200): No heartbeat from core client for 30 sec - exiting 08:31:57 (98200): No heartbeat from core client for 30 sec - exiting 08:31:58 (98200): No heartbeat from core client for 30 sec - exiting 08:31:59 (98200): No heartbeat from core client for 30 sec - exiting 08:32:00 (98200): No heartbeat from core client for 30 sec - exiting 08:32:01 (98200): No heartbeat from core client for 30 sec - exiting 08:33:47 (78576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:33:48 (78576): No heartbeat from core client for 30 sec - exiting 08:33:49 (78576): No heartbeat from core client for 30 sec - exiting 08:33:50 (78576): No heartbeat from core client for 30 sec - exiting 08:33:51 (78576): No heartbeat from core client for 30 sec - exiting 08:33:52 (78576): No heartbeat from core client for 30 sec - exiting 08:33:53 (78576): No heartbeat from core client for 30 sec - exiting 08:33:54 (78576): No heartbeat from core client for 30 sec - exiting 08:33:55 (78576): No heartbeat from core client for 30 sec - exiting 08:33:56 (78576): No heartbeat from core client for 30 sec - exiting 08:33:57 (78576): No heartbeat from core client for 30 sec - exiting 08:36:47 (104452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:39:40 (20208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:39:41 (20208): No heartbeat from core client for 30 sec - exiting 08:39:42 (20208): No heartbeat from core client for 30 sec - exiting 08:39:43 (20208): No heartbeat from core client for 30 sec - exiting 08:39:44 (20208): No heartbeat from core client for 30 sec - exiting 08:39:45 (20208): No heartbeat from core client for 30 sec - exiting 08:39:46 (20208): No heartbeat from core client for 30 sec - exiting 08:43:45 (97604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:43:46 (97604): No heartbeat from core client for 30 sec - exiting 08:43:47 (97604): No heartbeat from core client for 30 sec - exiting 08:43:48 (97604): No heartbeat from core client for 30 sec - exiting 08:43:49 (97604): No heartbeat from core client for 30 sec - exiting 08:43:50 (97604): No heartbeat from core client for 30 sec - exiting 08:44:23 (112012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:08 (93320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:53:15 (93348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:53:16 (93348): No heartbeat from core client for 30 sec - exiting 08:53:17 (93348): No heartbeat from core client for 30 sec - exiting 08:53:18 (93348): No heartbeat from core client for 30 sec - exiting 08:53:19 (93348): No heartbeat from core client for 30 sec - exiting 08:53:20 (93348): No heartbeat from core client for 30 sec - exiting 08:53:21 (93348): No heartbeat from core client for 30 sec - exiting 08:54:25 (109656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:54:26 (109656): No heartbeat from core client for 30 sec - exiting 08:54:27 (109656): No heartbeat from core client for 30 sec - exiting 08:54:28 (109656): No heartbeat from core client for 30 sec - exiting 08:54:29 (109656): No heartbeat from core client for 30 sec - exiting 08:54:30 (109656): No heartbeat from core client for 30 sec - exiting 08:54:31 (109656): No heartbeat from core client for 30 sec - exiting 08:54:32 (109656): No heartbeat from core client for 30 sec - exiting 08:54:33 (109656): No heartbeat from core client for 30 sec - exiting 08:54:34 (109656): No heartbeat from core client for 30 sec - exiting 08:54:35 (109656): No heartbeat from core client for 30 sec - exiting 08:57:55 (109808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:57:56 (109808): No heartbeat from core client for 30 sec - exiting 08:57:57 (109808): No heartbeat from core client for 30 sec - exiting 08:57:58 (109808): No heartbeat from core client for 30 sec - exiting 08:57:59 (109808): No heartbeat from core client for 30 sec - exiting 08:58:00 (109808): No heartbeat from core client for 30 sec - exiting 08:58:01 (109808): No heartbeat from core client for 30 sec - exiting 08:58:02 (109808): No heartbeat from core client for 30 sec - exiting 08:58:03 (109808): No heartbeat from core client for 30 sec - exiting 08:58:04 (109808): No heartbeat from core client for 30 sec - exiting 08:58:05 (109808): No heartbeat from core client for 30 sec - exiting 09:00:24 (86828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:25 (86828): No heartbeat from core client for 30 sec - exiting 09:05:44 (24180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:45 (24180): No heartbeat from core client for 30 sec - exiting 09:05:46 (24180): No heartbeat from core client for 30 sec - exiting 09:05:47 (24180): No heartbeat from core client for 30 sec - exiting 09:05:48 (24180): No heartbeat from core client for 30 sec - exiting 09:05:49 (24180): No heartbeat from core client for 30 sec - exiting 09:07:08 (93112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:09:20 (100736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:09:21 (100736): No heartbeat from core client for 30 sec - exiting 09:09:22 (100736): No heartbeat from core client for 30 sec - exiting 09:09:23 (100736): No heartbeat from core client for 30 sec - exiting 09:09:24 (100736): No heartbeat from core client for 30 sec - exiting 09:09:25 (100736): No heartbeat from core client for 30 sec - exiting 09:09:58 (93872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:10:51 (64528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:10:52 (64528): No heartbeat from core client for 30 sec - exiting 09:10:53 (64528): No heartbeat from core client for 30 sec - exiting 09:10:54 (64528): No heartbeat from core client for 30 sec - exiting 09:10:55 (64528): No heartbeat from core client for 30 sec - exiting 09:10:56 (64528): No heartbeat from core client for 30 sec - exiting 09:10:57 (64528): No heartbeat from core client for 30 sec - exiting 09:10:58 (64528): No heartbeat from core client for 30 sec - exiting 09:10:59 (64528): No heartbeat from core client for 30 sec - exiting 09:11:00 (64528): No heartbeat from core client for 30 sec - exiting 09:11:01 (64528): No heartbeat from core client for 30 sec - exiting 09:14:42 (101040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:15:17 (90036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:22:29 (88556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:24:04 (90980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:24:05 (90980): No heartbeat from core client for 30 sec - exiting 09:24:06 (90980): No heartbeat from core client for 30 sec - exiting 09:24:07 (90980): No heartbeat from core client for 30 sec - exiting 09:24:08 (90980): No heartbeat from core client for 30 sec - exiting 09:29:59 (93888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:30:00 (93888): No heartbeat from core client for 30 sec - exiting 09:30:01 (93888): No heartbeat from core client for 30 sec - exiting 09:33:02 (80920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:33:03 (80920): No heartbeat from core client for 30 sec - exiting 09:33:04 (80920): No heartbeat from core client for 30 sec - exiting 09:33:05 (80920): No heartbeat from core client for 30 sec - exiting 09:33:06 (80920): No heartbeat from core client for 30 sec - exiting 09:33:07 (80920): No heartbeat from core client for 30 sec - exiting 09:33:08 (80920): No heartbeat from core client for 30 sec - exiting 09:33:09 (80920): No heartbeat from core client for 30 sec - exiting 09:33:10 (80920): No heartbeat from core client for 30 sec - exiting 09:33:11 (80920): No heartbeat from core client for 30 sec - exiting 09:33:12 (80920): No heartbeat from core client for 30 sec - exiting 09:36:20 (23800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:36:21 (23800): No heartbeat from core client for 30 sec - exiting 09:36:22 (23800): No heartbeat from core client for 30 sec - exiting 09:36:23 (23800): No heartbeat from core client for 30 sec - exiting 09:36:24 (23800): No heartbeat from core client for 30 sec - exiting 09:36:25 (23800): No heartbeat from core client for 30 sec - exiting 09:36:26 (23800): No heartbeat from core client for 30 sec - exiting 09:36:27 (23800): No heartbeat from core client for 30 sec - exiting 09:36:28 (23800): No heartbeat from core client for 30 sec - exiting 09:36:29 (23800): No heartbeat from core client for 30 sec - exiting 09:36:30 (23800): No heartbeat from core client for 30 sec - exiting 09:37:58 (93616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:40:44 (26088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:40:45 (26088): No heartbeat from core client for 30 sec - exiting 09:40:46 (26088): No heartbeat from core client for 30 sec - exiting 09:40:47 (26088): No heartbeat from core client for 30 sec - exiting 09:40:48 (26088): No heartbeat from core client for 30 sec - exiting 09:41:23 (100180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:44:46 (60984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:44:47 (60984): No heartbeat from core client for 30 sec - exiting 09:44:48 (60984): No heartbeat from core client for 30 sec - exiting 09:44:49 (60984): No heartbeat from core client for 30 sec - exiting 09:44:50 (60984): No heartbeat from core client for 30 sec - exiting 09:44:51 (60984): No heartbeat from core client for 30 sec - exiting 09:44:52 (60984): No heartbeat from core client for 30 sec - exiting 09:46:30 (91272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:46:31 (91272): No heartbeat from core client for 30 sec - exiting 09:46:32 (91272): No heartbeat from core client for 30 sec - exiting 09:46:33 (91272): No heartbeat from core client for 30 sec - exiting 09:49:15 (95764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:49:16 (95764): No heartbeat from core client for 30 sec - exiting 09:49:17 (95764): No heartbeat from core client for 30 sec - exiting 09:49:18 (95764): No heartbeat from core client for 30 sec - exiting 09:49:19 (95764): No heartbeat from core client for 30 sec - exiting 09:49:20 (95764): No heartbeat from core client for 30 sec - exiting 09:49:21 (95764): No heartbeat from core client for 30 sec - exiting 09:49:22 (95764): No heartbeat from core client for 30 sec - exiting 09:49:23 (95764): No heartbeat from core client for 30 sec - exiting 09:49:24 (95764): No heartbeat from core client for 30 sec - exiting 09:49:25 (95764): No heartbeat from core client for 30 sec - exiting 09:50:51 (94232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:50:52 (94232): No heartbeat from core client for 30 sec - exiting 09:50:53 (94232): No heartbeat from core client for 30 sec - exiting 09:50:54 (94232): No heartbeat from core client for 30 sec - exiting 09:50:55 (94232): No heartbeat from core client for 30 sec - exiting 09:50:56 (94232): No heartbeat from core client for 30 sec - exiting 09:50:57 (94232): No heartbeat from core client for 30 sec - exiting 09:50:58 (94232): No heartbeat from core client for 30 sec - exiting 09:50:59 (94232): No heartbeat from core client for 30 sec - exiting 09:51:00 (94232): No heartbeat from core client for 30 sec - exiting 09:51:01 (94232): No heartbeat from core client for 30 sec - exiting 09:54:07 (85144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:41 (78580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:58:49 (89124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:59:28 (51340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:00:13 (95440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:36 (91696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:04:18 (102312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:05:32 (90148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:19:42 (104408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:05 (98452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:32:10 (93048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:37:47 (97328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:40:29 (103736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:45:49 (94296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:50:17 (90880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:53:20 (97968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:54:17 (88708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:54:56 (107524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:55:53 (91536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:59:09 (100720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:59:51 (91652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:03:10 (96880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:03:48 (92396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:04:35 (86668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:05:17 (80824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:07:46 (89924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:08:49 (102584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:11:52 (99920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:12:39 (99808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:13:30 (103596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:14:13 (91640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:17:14 (90340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:17:54 (95520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:18:42 (91132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:20:10 (96148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:24:43 (104476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:38:09 (103196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:42:04 (97184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:33:26 (96924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:35:27 (112712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:38:59 (96592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:42:16 (113300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:44:49 (113132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:47:47 (87532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:51:39 (99712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=112844, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=124396, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2468, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2468, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2468, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2468, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Apr 2013 21:30:46	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	311,040	273,364	0.8789
23 Apr 2013 13:45:07	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	285,120	249,980	0.8768
22 Apr 2013 20:48:44	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	259,200	226,926	0.8755
22 Apr 2013 14:00:29	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	233,280	203,663	0.8730
22 Apr 2013 01:11:13	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	207,360	180,657	0.8712
20 Apr 2013 23:59:27	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	181,440	157,028	0.8655
20 Apr 2013 09:19:06	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	155,520	133,995	0.8616
19 Apr 2013 21:29:52	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	129,600	111,251	0.8584
19 Apr 2013 13:00:04	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	103,680	88,635	0.8549
18 Apr 2013 21:04:19	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	77,760	67,461	0.8676
18 Apr 2013 13:55:26	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	51,840	45,396	0.8757
18 Apr 2013 02:53:46	1239995	15728291	hadcm3n_zcz8_1960_40_008349795_0	25,920	22,857	0.8818