Task 15492353

Name	hadcm3n_3b9i_1940_40_008263571_0
Workunit	8418695
Created	21 Dec 2012, 5:47:19 UTC
Sent	23 Dec 2012, 10:02:16 UTC
Report deadline	24 Mar 2013, 17:29:27 UTC
Received	25 Feb 2013, 22:53:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1016874
Run time	23 days 4 hours 10 min 5 sec
CPU time	20 days 21 hours 35 min 17 sec
Validate state	Invalid
Credit	9,953.28
Device peak FLOPS	1.92 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 15:12:35 (3776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:12:36 (3776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 11:18:33 (3864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:19:49 (4384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:19:51 (4384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:02:35 (2540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:02:39 (2540): No heartbeat from core client for 30 sec - exiting 16:02:41 (2540): No heartbeat from core client for 30 sec - exiting 16:02:42 (2540): No heartbeat from core client for 30 sec - exiting 16:02:43 (2540): No heartbeat from core client for 30 sec - exiting 16:02:44 (2540): No heartbeat from core client for 30 sec - exiting 16:04:53 (2992): No heartbeat from core client for 30 sec - exiting 16:04:55 (2992): No heartbeat from core client for 30 sec - exiting 16:04:56 (2992): No heartbeat from core client for 30 sec - exiting 16:04:57 (2992): No heartbeat from core client for 30 sec - exiting 16:04:58 (2992): No heartbeat from core client for 30 sec - exiting 16:04:59 (2992): No heartbeat from core client for 30 sec - exiting 16:05:00 (2992): No heartbeat from core client for 30 sec - exiting 16:05:01 (2992): No heartbeat from core client for 30 sec - exiting 16:05:02 (2992): No heartbeat from core client for 30 sec - exiting 16:05:03 (2992): No heartbeat from core client for 30 sec - exiting 16:05:04 (2992): No heartbeat from core client for 30 sec - exiting 16:05:05 (2992): No heartbeat from core client for 30 sec - exiting 16:05:06 (2992): No heartbeat from core client for 30 sec - exiting 16:05:07 (2992): No heartbeat from core client for 30 sec - exiting 16:05:08 (2992): No heartbeat from core client for 30 sec - exiting 16:05:09 (2992): No heartbeat from core client for 30 sec - exiting 16:05:10 (2992): No heartbeat from core client for 30 sec - exiting 16:05:11 (2992): No heartbeat from core client for 30 sec - exiting 16:05:12 (2992): No heartbeat from core client for 30 sec - exiting 16:05:13 (2992): No heartbeat from core client for 30 sec - exiting 16:05:14 (2992): No heartbeat from core client for 30 sec - exiting 16:05:15 (2992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:09:42 (5164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:09:44 (5164): No heartbeat from core client for 30 sec - exiting 16:09:46 (5164): No heartbeat from core client for 30 sec - exiting 16:11:02 (3312): No heartbeat from core client for 30 sec - exiting 16:11:03 (3312): No heartbeat from core client for 30 sec - exiting 16:11:04 (3312): No heartbeat from core client for 30 sec - exiting 16:11:05 (3312): No heartbeat from core client for 30 sec - exiting 16:11:06 (3312): No heartbeat from core client for 30 sec - exiting 16:11:07 (3312): No heartbeat from core client for 30 sec - exiting 16:11:08 (3312): No heartbeat from core client for 30 sec - exiting 16:11:09 (3312): No heartbeat from core client for 30 sec - exiting 16:11:10 (3312): No heartbeat from core client for 30 sec - exiting 16:11:11 (3312): No heartbeat from core client for 30 sec - exiting 16:11:12 (3312): No heartbeat from core client for 30 sec - exiting 16:11:13 (3312): No heartbeat from core client for 30 sec - exiting 16:11:14 (3312): No heartbeat from core client for 30 sec - exiting 16:11:15 (3312): No heartbeat from core client for 30 sec - exiting 16:11:16 (3312): No heartbeat from core client for 30 sec - exiting 16:11:17 (3312): No heartbeat from core client for 30 sec - exiting 16:11:18 (3312): No heartbeat from core client for 30 sec - exiting 16:11:19 (3312): No heartbeat from core client for 30 sec - exiting 16:11:20 (3312): No heartbeat from core client for 30 sec - exiting 16:11:21 (3312): No heartbeat from core client for 30 sec - exiting 16:11:22 (3312): No heartbeat from core client for 30 sec - exiting 16:11:23 (3312): No heartbeat from core client for 30 sec - exiting 16:11:24 (3312): No heartbeat from core client for 30 sec - exiting 16:11:25 (3312): No heartbeat from core client for 30 sec - exiting 16:11:26 (3312): No heartbeat from core client for 30 sec - exiting 16:11:27 (3312): No heartbeat from core client for 30 sec - exiting 16:11:28 (3312): No heartbeat from core client for 30 sec - exiting 16:11:29 (3312): No heartbeat from core client for 30 sec - exiting 16:11:30 (3312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:13:27 (992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:44:41 (4176): No heartbeat from core client for 30 sec - exiting 11:44:43 (4176): No heartbeat from core client for 30 sec - exiting 11:44:44 (4176): No heartbeat from core client for 30 sec - exiting 11:44:45 (4176): No heartbeat from core client for 30 sec - exiting 11:44:46 (4176): No heartbeat from core client for 30 sec - exiting 11:44:47 (4176): No heartbeat from core client for 30 sec - exiting 11:44:48 (4176): No heartbeat from core client for 30 sec - exiting 11:44:49 (4176): No heartbeat from core client for 30 sec - exiting 11:44:50 (4176): No heartbeat from core client for 30 sec - exiting 11:44:51 (4176): No heartbeat from core client for 30 sec - exiting 11:44:52 (4176): No heartbeat from core client for 30 sec - exiting 11:44:53 (4176): No heartbeat from core client for 30 sec - exiting 11:44:54 (4176): No heartbeat from core client for 30 sec - exiting 11:44:55 (4176): No heartbeat from core client for 30 sec - exiting 11:44:56 (4176): No heartbeat from core client for 30 sec - exiting 11:44:57 (4176): No heartbeat from core client for 30 sec - exiting 11:44:58 (4176): No heartbeat from core client for 30 sec - exiting 11:44:59 (4176): No heartbeat from core client for 30 sec - exiting 11:45:00 (4176): No heartbeat from core client for 30 sec - exiting 11:45:01 (4176): No heartbeat from core client for 30 sec - exiting 11:45:02 (4176): No heartbeat from core client for 30 sec - exiting 11:45:03 (4176): No heartbeat from core client for 30 sec - exiting 11:45:04 (4176): No heartbeat from core client for 30 sec - exiting 11:45:05 (4176): No heartbeat from core client for 30 sec - exiting 11:45:06 (4176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:45:07 (4176): No heartbeat from core client for 30 sec - exiting 11:45:08 (4176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 23:20:12 (2276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3b9iko.pje5c10 Error converting file to netcdf: dataout/3b9iko.pie5c10 Error converting file to netcdf: dataout/3b9iko.pfe5c10 Error converting file to netcdf: dataout/3b9ika.phe5c10 Error converting file to netcdf: dataout/3b9ika.pge5c10 Error converting file to netcdf: dataout/3b9ika.pee5c10 Error converting file to netcdf: dataout/3b9ika.pde5c10 CPDN Monitor - Quit request from BOINC... 20:02:04 (3300): No heartbeat from core client for 30 sec - exiting 20:02:05 (3300): No heartbeat from core client for 30 sec - exiting 20:02:06 (3300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:17:02 (2392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:15:53 (4436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:28:38 (3848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:14:37 (2172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:13:23 (4596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:12:08 (1212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:11:04 (4920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:14:07 (1344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:45:52 (2408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:44:48 (4384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:12:40 (704): No heartbeat from core client for 30 sec - exiting 13:12:41 (704): No heartbeat from core client for 30 sec - exiting 13:12:42 (704): No heartbeat from core client for 30 sec - exiting 13:12:43 (704): No heartbeat from core client for 30 sec - exiting 13:12:44 (704): No heartbeat from core client for 30 sec - exiting 13:12:45 (704): No heartbeat from core client for 30 sec - exiting 13:12:46 (704): No heartbeat from core client for 30 sec - exiting 13:12:47 (704): No heartbeat from core client for 30 sec - exiting 13:12:48 (704): No heartbeat from core client for 30 sec - exiting 13:12:49 (704): No heartbeat from core client for 30 sec - exiting 13:12:50 (704): No heartbeat from core client for 30 sec - exiting 13:12:51 (704): No heartbeat from core client for 30 sec - exiting 13:12:52 (704): No heartbeat from core client for 30 sec - exiting 13:12:53 (704): No heartbeat from core client for 30 sec - exiting 13:12:54 (704): No heartbeat from core client for 30 sec - exiting 13:12:55 (704): No heartbeat from core client for 30 sec - exiting 13:12:56 (704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:57 (704): No heartbeat from core client for 30 sec - exiting 13:12:58 (704): No heartbeat from core client for 30 sec - exiting 13:12:59 (704): No heartbeat from core client for 30 sec - exiting 13:13:00 (704): No heartbeat from core client for 30 sec - exiting 13:13:01 (704): No heartbeat from core client for 30 sec - exiting 13:13:02 (704): No heartbeat from core client for 30 sec - exiting 13:13:03 (704): No heartbeat from core client for 30 sec - exiting 13:13:04 (704): No heartbeat from core client for 30 sec - exiting 13:13:05 (704): No heartbeat from core client for 30 sec - exiting 13:13:06 (704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:51:02 (2084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:51:03 (2084): No heartbeat from core client for 30 sec - exiting 23:51:59 (5492): No heartbeat from core client for 30 sec - exiting 23:52:00 (5492): No heartbeat from core client for 30 sec - exiting 23:52:01 (5492): No heartbeat from core client for 30 sec - exiting 23:52:02 (5492): No heartbeat from core client for 30 sec - exiting 23:52:03 (5492): No heartbeat from core client for 30 sec - exiting 23:52:04 (5492): No heartbeat from core client for 30 sec - exiting 23:52:05 (5492): No heartbeat from core client for 30 sec - exiting 23:52:06 (5492): No heartbeat from core client for 30 sec - exiting 23:52:07 (5492): No heartbeat from core client for 30 sec - exiting 23:52:08 (5492): No heartbeat from core client for 30 sec - exiting 23:52:09 (5492): No heartbeat from core client for 30 sec - exiting 23:52:10 (5492): No heartbeat from core client for 30 sec - exiting 23:52:11 (5492): No heartbeat from core client for 30 sec - exiting 23:52:12 (5492): No heartbeat from core client for 30 sec - exiting 23:52:13 (5492): No heartbeat from core client for 30 sec - exiting 23:52:14 (5492): No heartbeat from core client for 30 sec - exiting 23:52:15 (5492): No heartbeat from core client for 30 sec - exiting 23:52:16 (5492): No heartbeat from core client for 30 sec - exiting 23:52:17 (5492): No heartbeat from core client for 30 sec - exiting 23:52:18 (5492): No heartbeat from core client for 30 sec - exiting 23:52:20 (5492): No heartbeat from core client for 30 sec - exiting 23:52:21 (5492): No heartbeat from core client for 30 sec - exiting 23:52:22 (5492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:06:04 (5572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:06:51 (2600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:16:05 (6096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:01:36 (4296): No heartbeat from core client for 30 sec - exiting 10:01:37 (4296): No heartbeat from core client for 30 sec - exiting 10:01:38 (4296): No heartbeat from core client for 30 sec - exiting 10:01:39 (4296): No heartbeat from core client for 30 sec - exiting 10:01:40 (4296): No heartbeat from core client for 30 sec - exiting 10:01:41 (4296): No heartbeat from core client for 30 sec - exiting 10:01:42 (4296): No heartbeat from core client for 30 sec - exiting 10:01:43 (4296): No heartbeat from core client for 30 sec - exiting 10:01:44 (4296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:10:47 (2948): No heartbeat from core client for 30 sec - exiting 19:10:48 (2948): No heartbeat from core client for 30 sec - exiting 19:10:49 (2948): No heartbeat from core client for 30 sec - exiting 19:10:50 (2948): No heartbeat from core client for 30 sec - exiting 19:10:51 (2948): No heartbeat from core client for 30 sec - exiting 19:10:52 (2948): No heartbeat from core client for 30 sec - exiting 19:10:53 (2948): No heartbeat from core client for 30 sec - exiting 19:10:54 (2948): No heartbeat from core client for 30 sec - exiting 19:10:55 (2948): No heartbeat from core client for 30 sec - exiting 19:10:56 (2948): No heartbeat from core client for 30 sec - exiting 19:10:57 (2948): No heartbeat from core client for 30 sec - exiting 19:10:58 (2948): No heartbeat from core client for 30 sec - exiting 19:10:59 (2948): No heartbeat from core client for 30 sec - exiting 19:11:00 (2948): No heartbeat from core client for 30 sec - exiting 19:11:01 (2948): No heartbeat from core client for 30 sec - exiting 19:11:02 (2948): No heartbeat from core client for 30 sec - exiting 19:11:03 (2948): No heartbeat from core client for 30 sec - exiting 19:11:04 (2948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:11:05 (2948): No heartbeat from core client for 30 sec - exiting 19:11:06 (2948): No heartbeat from core client for 30 sec - exiting 19:11:07 (2948): No heartbeat from core client for 30 sec - exiting 19:11:09 (2948): No heartbeat from core client for 30 sec - exiting 19:11:11 (2948): No heartbeat from core client for 30 sec - exiting 19:11:12 (2948): No heartbeat from core client for 30 sec - exiting 19:11:13 (2948): No heartbeat from core client for 30 sec - exiting 19:11:14 (2948): No heartbeat from core client for 30 sec - exiting 19:11:15 (2948): No heartbeat from core client for 30 sec - exiting 19:20:12 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:20:14 (5720): No heartbeat from core client for 30 sec - exiting 19:20:15 (5720): No heartbeat from core client for 30 sec - exiting 19:21:19 (4212): No heartbeat from core client for 30 sec - exiting 19:21:20 (4212): No heartbeat from core client for 30 sec - exiting 19:21:21 (4212): No heartbeat from core client for 30 sec - exiting 19:21:22 (4212): No heartbeat from core client for 30 sec - exiting 19:21:23 (4212): No heartbeat from core client for 30 sec - exiting 19:21:24 (4212): No heartbeat from core client for 30 sec - exiting 19:21:25 (4212): No heartbeat from core client for 30 sec - exiting 19:21:26 (4212): No heartbeat from core client for 30 sec - exiting 19:21:27 (4212): No heartbeat from core client for 30 sec - exiting 19:21:28 (4212): No heartbeat from core client for 30 sec - exiting 19:21:29 (4212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:21:30 (4212): No heartbeat from core client for 30 sec - exiting 19:30:26 (1700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:30:28 (1700): No heartbeat from core client for 30 sec - exiting 19:30:29 (1700): No heartbeat from core client for 30 sec - exiting 19:31:32 (4632): No heartbeat from core client for 30 sec - exiting 19:31:33 (4632): No heartbeat from core client for 30 sec - exiting 19:31:34 (4632): No heartbeat from core client for 30 sec - exiting 19:31:35 (4632): No heartbeat from core client for 30 sec - exiting 19:31:36 (4632): No heartbeat from core client for 30 sec - exiting 19:31:37 (4632): No heartbeat from core client for 30 sec - exiting 19:31:38 (4632): No heartbeat from core client for 30 sec - exiting 19:31:39 (4632): No heartbeat from core client for 30 sec - exiting 19:31:40 (4632): No heartbeat from core client for 30 sec - exiting 19:31:41 (4632): No heartbeat from core client for 30 sec - exiting 19:31:42 (4632): No heartbeat from core client for 30 sec - exiting 19:31:43 (4632): No heartbeat from core client for 30 sec - exiting 19:31:44 (4632): No heartbeat from core client for 30 sec - exiting 19:31:45 (4632): No heartbeat from core client for 30 sec - exiting 19:31:46 (4632): No heartbeat from core client for 30 sec - exiting 19:31:47 (4632): No heartbeat from core client for 30 sec - exiting 19:31:48 (4632): No heartbeat from core client for 30 sec - exiting 19:31:49 (4632): No heartbeat from core client for 30 sec - exiting 19:31:50 (4632): No heartbeat from core client for 30 sec - exiting 19:31:51 (4632): No heartbeat from core client for 30 sec - exiting 19:31:52 (4632): No heartbeat from core client for 30 sec - exiting 19:31:53 (4632): No heartbeat from core client for 30 sec - exiting 19:31:54 (4632): No heartbeat from core client for 30 sec - exiting 19:31:55 (4632): No heartbeat from core client for 30 sec - exiting 19:31:56 (4632): No heartbeat from core client for 30 sec - exiting 19:31:57 (4632): No heartbeat from core client for 30 sec - exiting 19:31:58 (4632): No heartbeat from core client for 30 sec - exiting 19:31:59 (4632): No heartbeat from core client for 30 sec - exiting 19:32:00 (4632): No heartbeat from core client for 30 sec - exiting 19:32:01 (4632): No heartbeat from core client for 30 sec - exiting 19:32:02 (4632): No heartbeat from core client for 30 sec - exiting 19:32:03 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:32:04 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 11:25:00 (1672): No heartbeat from core client for 30 sec - exiting 11:25:01 (1672): No heartbeat from core client for 30 sec - exiting 11:25:02 (1672): No heartbeat from core client for 30 sec - exiting 11:25:03 (1672): No heartbeat from core client for 30 sec - exiting 11:25:04 (1672): No heartbeat from core client for 30 sec - exiting 11:25:05 (1672): No heartbeat from core client for 30 sec - exiting 11:25:06 (1672): No heartbeat from core client for 30 sec - exiting 11:25:07 (1672): No heartbeat from core client for 30 sec - exiting 11:25:08 (1672): No heartbeat from core client for 30 sec - exiting 11:25:09 (1672): No heartbeat from core client for 30 sec - exiting 11:25:10 (1672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:18:06 (3800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:48 (5304): No heartbeat from core client for 30 sec - exiting 22:34:49 (5304): No heartbeat from core client for 30 sec - exiting 22:34:50 (5304): No heartbeat from core client for 30 sec - exiting 22:34:51 (5304): No heartbeat from core client for 30 sec - exiting 22:34:52 (5304): No heartbeat from core client for 30 sec - exiting 22:34:53 (5304): No heartbeat from core client for 30 sec - exiting 22:34:54 (5304): No heartbeat from core client for 30 sec - exiting 22:34:55 (5304): No heartbeat from core client for 30 sec - exiting 22:34:56 (5304): No heartbeat from core client for 30 sec - exiting 22:34:57 (5304): No heartbeat from core client for 30 sec - exiting 22:34:58 (5304): No heartbeat from core client for 30 sec - exiting 22:35:00 (5304): No heartbeat from core client for 30 sec - exiting 22:35:01 (5304): No heartbeat from core client for 30 sec - exiting 22:35:02 (5304): No heartbeat from core client for 30 sec - exiting 22:35:03 (5304): No heartbeat from core client for 30 sec - exiting 22:35:04 (5304): No heartbeat from core client for 30 sec - exiting 22:35:05 (5304): No heartbeat from core client for 30 sec - exiting 22:35:06 (5304): No heartbeat from core client for 30 sec - exiting 22:35:07 (5304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3b9iko.pjg1c10 Error converting file to netcdf: dataout/3b9iko.pig1c10 Error converting file to netcdf: dataout/3b9iko.pfg1c10 Error converting file to netcdf: dataout/3b9ika.phg1c10 Error converting file to netcdf: dataout/3b9ika.pgg1c10 Error converting file to netcdf: dataout/3b9ika.peg1c10 Error converting file to netcdf: dataout/3b9ika.pdg1c10 CPDN Monitor - Quit request from BOINC... 19:33:01 (3932): No heartbeat from core client for 30 sec - exiting 19:33:03 (3932): No heartbeat from core client for 30 sec - exiting 19:33:04 (3932): No heartbeat from core client for 30 sec - exiting 19:33:05 (3932): No heartbeat from core client for 30 sec - exiting 19:33:06 (3932): No heartbeat from core client for 30 sec - exiting 19:33:07 (3932): No heartbeat from core client for 30 sec - exiting 19:33:08 (3932): No heartbeat from core client for 30 sec - exiting 19:33:09 (3932): No heartbeat from core client for 30 sec - exiting 19:33:10 (3932): No heartbeat from core client for 30 sec - exiting 19:33:11 (3932): No heartbeat from core client for 30 sec - exiting 19:33:12 (3932): No heartbeat from core client for 30 sec - exiting 19:33:14 (3932): No heartbeat from core client for 30 sec - exiting 19:33:15 (3932): No heartbeat from core client for 30 sec - exiting 19:33:16 (3932): No heartbeat from core client for 30 sec - exiting 19:33:17 (3932): No heartbeat from core client for 30 sec - exiting 19:33:18 (3932): No heartbeat from core client for 30 sec - exiting 19:33:20 (3932): No heartbeat from core client for 30 sec - exiting 19:33:21 (3932): No heartbeat from core client for 30 sec - exiting 19:33:23 (3932): No heartbeat from core client for 30 sec - exiting 19:33:24 (3932): No heartbeat from core client for 30 sec - exiting 19:33:26 (3932): No heartbeat from core client for 30 sec - exiting 19:33:27 (3932): No heartbeat from core client for 30 sec - exiting 19:33:28 (3932): No heartbeat from core client for 30 sec - exiting 19:33:29 (3932): No heartbeat from core client for 30 sec - exiting 19:33:30 (3932): No heartbeat from core client for 30 sec - exiting 19:33:32 (3932): No heartbeat from core client for 30 sec - exiting 19:33:33 (3932): No heartbeat from core client for 30 sec - exiting 19:33:34 (3932): No heartbeat from core client for 30 sec - exiting 19:33:35 (3932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:48:15 (724): No heartbeat from core client for 30 sec - exiting 21:48:18 (724): No heartbeat from core client for 30 sec - exiting 21:48:19 (724): No heartbeat from core client for 30 sec - exiting 21:48:21 (724): No heartbeat from core client for 30 sec - exiting 21:48:22 (724): No heartbeat from core client for 30 sec - exiting 21:48:24 (724): No heartbeat from core client for 30 sec - exiting 21:48:25 (724): No heartbeat from core client for 30 sec - exiting 21:48:28 (724): No heartbeat from core client for 30 sec - exiting 21:48:29 (724): No heartbeat from core client for 30 sec - exiting 21:48:30 (724): No heartbeat from core client for 30 sec - exiting 21:48:31 (724): No heartbeat from core client for 30 sec - exiting 21:48:33 (724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:48:34 (724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 21:55:55 (5116): No heartbeat from core client for 30 sec - exiting 21:55:56 (5116): No heartbeat from core client for 30 sec - exiting 21:55:57 (5116): No heartbeat from core client for 30 sec - exiting 21:55:58 (5116): No heartbeat from core client for 30 sec - exiting 21:55:59 (5116): No heartbeat from core client for 30 sec - exiting 21:56:00 (5116): No heartbeat from core client for 30 sec - exiting 21:56:01 (5116): No heartbeat from core client for 30 sec - exiting 21:56:02 (5116): No heartbeat from core client for 30 sec - exiting 21:56:03 (5116): No heartbeat from core client for 30 sec - exiting 21:56:04 (5116): No heartbeat from core client for 30 sec - exiting 21:56:05 (5116): No heartbeat from core client for 30 sec - exiting 21:56:06 (5116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:18:39 (2668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:34:35 (1352): No heartbeat from core client for 30 sec - exiting 23:34:36 (1352): No heartbeat from core client for 30 sec - exiting 23:34:37 (1352): No heartbeat from core client for 30 sec - exiting 23:34:38 (1352): No heartbeat from core client for 30 sec - exiting 23:34:39 (1352): No heartbeat from core client for 30 sec - exiting 23:34:40 (1352): No heartbeat from core client for 30 sec - exiting 23:34:41 (1352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:19:39 (3144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 01:36:32 (2280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:35:17 (2584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:34:01 (3908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:32:51 (4472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:31:35 (2296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:30:19 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:29:02 (2916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:57 (2300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:07 (5764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:26:54 (4692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:25:51 (4032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:48 (2892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:23:43 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:19:59 (5940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4768, iMonCtr=1 Model crash detected, will try to restart... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4768, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Feb 2013 18:59:58	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	829,440	1,788,261	2.1560
23 Feb 2013 03:31:01	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	803,520	1,734,731	2.1589
22 Feb 2013 11:22:29	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	777,600	1,680,806	2.1615
21 Feb 2013 19:51:54	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	751,680	1,627,162	2.1647
21 Feb 2013 04:33:44	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	725,760	1,573,457	2.1680
20 Feb 2013 13:05:11	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	699,840	1,519,802	2.1716
19 Feb 2013 21:24:50	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	673,920	1,465,306	2.1743
19 Feb 2013 05:22:31	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	648,000	1,409,065	2.1745
17 Feb 2013 17:05:59	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	622,080	1,352,528	2.1742
17 Feb 2013 01:00:26	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	596,160	1,296,399	2.1746
16 Feb 2013 07:58:45	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	570,240	1,239,258	2.1732
10 Feb 2013 22:43:30	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	544,320	1,182,452	2.1723
08 Feb 2013 22:54:11	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	518,400	1,126,159	2.1724
28 Jan 2013 21:16:54	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	492,480	1,069,416	2.1715
24 Jan 2013 23:58:32	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	466,560	1,014,984	2.1755
23 Jan 2013 10:33:31	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	440,640	961,253	2.1815
22 Jan 2013 19:33:18	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	414,720	907,531	2.1883
22 Jan 2013 04:09:39	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	388,800	854,076	2.1967
21 Jan 2013 12:55:11	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	362,880	800,522	2.2060
20 Jan 2013 20:47:33	1016874	15492353	hadcm3n_3b9i_1940_40_008263571_0	336,960	747,074	2.2171