Task 13548882

Name	hadcm3n_y9f6_1900_40_007521597_0
Workunit	7719072
Created	28 Oct 2011, 13:13:44 UTC
Sent	2 Nov 2011, 9:24:40 UTC
Report deadline	1 Feb 2012, 16:51:51 UTC
Received	3 Jan 2012, 23:59:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1064035
Run time	11 days 19 hours 15 min 44 sec
CPU time	9 days 5 hours 2 min 54 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.72 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 02:24:37 (8032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:58:26 (8948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:16:12 (10337): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:47 (11532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:55:19 (12252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:10 (15818): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:44:14 (16504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:00:25 (17430): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:13:40 (17568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:13:41 (20304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:13:30 (21335): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:46:41 (28436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:28:49 (31145): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:42:07 (31274): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:17:01 (31382): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:44:37 (31672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:55:24 (31905): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:54:05 (31999): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:05:42 (32479): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:38:40 (32577): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:49:27 (32856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:23:31 (32943): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:52:57 (33232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:06:51 (33474): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:18:12 (33585): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:48:23 (33679): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:01:42 (33932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:33:02 (34047): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:33:23 (34047): No heartbeat from core client for 30 sec - exiting 01:43:40 (34307): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:54:54 (34390): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:19:19 (34487): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:48:09 (34690): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:21 (34922): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:24:11 (35106): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:42:42 (35324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:43:20 (35324): No heartbeat from core client for 30 sec - exiting 04:17:25 (35473): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:37:31 (35757): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:59:23 (35926): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:10:28 (36109): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:23:12 (36204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:33:47 (36310): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:02:21 (36398): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:34:02 (36636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:30:24 (36897): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:42:04 (37368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:54:25 (37462): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:32:58 (37568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:44:21 (37890): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:06:16 (37990): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:26:04 (38170): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:39:50 (38332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:01:09 (38447): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:13:33 (43894): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:13:19 (51773): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:03:15 (52147): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:34:16 (52697): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:41:54 (59790): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:08 (59946): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:14:08 (60835): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:54:35 (62326): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:02:48 (81594): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:19 (81594): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:18:10 (90317): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:40:30 (93345): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:05:13 (1677): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:00:39 (2624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:47:04 (5353): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:29:49 (16755): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:03:54 (37834): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:23:02 (38152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:48:41 (44915): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:42 (44915): No heartbeat from core client for 30 sec - exiting 10:48:43 (44915): No heartbeat from core client for 30 sec - exiting 10:48:44 (44915): No heartbeat from core client for 30 sec - exiting 10:48:45 (44915): No heartbeat from core client for 30 sec - exiting 10:48:46 (44915): No heartbeat from core client for 30 sec - exiting 10:48:47 (44915): No heartbeat from core client for 30 sec - exiting 10:48:48 (44915): No heartbeat from core client for 30 sec - exiting 10:48:49 (44915): No heartbeat from core client for 30 sec - exiting 10:48:50 (44915): No heartbeat from core client for 30 sec - exiting 10:48:51 (44915): No heartbeat from core client for 30 sec - exiting 10:48:52 (44915): No heartbeat from core client for 30 sec - exiting 10:48:53 (44915): No heartbeat from core client for 30 sec - exiting 10:48:54 (44915): No heartbeat from core client for 30 sec - exiting 10:48:55 (44915): No heartbeat from core client for 30 sec - exiting 10:48:56 (44915): No heartbeat from core client for 30 sec - exiting 10:48:57 (44915): No heartbeat from core client for 30 sec - exiting 10:48:58 (44915): No heartbeat from core client for 30 sec - exiting 10:48:59 (44915): No heartbeat from core client for 30 sec - exiting 10:49:00 (44915): No heartbeat from core client for 30 sec - exiting 10:49:01 (44915): No heartbeat from core client for 30 sec - exiting 10:49:02 (44915): No heartbeat from core client for 30 sec - exiting 10:49:03 (44915): No heartbeat from core client for 30 sec - exiting 10:49:04 (44915): No heartbeat from core client for 30 sec - exiting 10:49:05 (44915): No heartbeat from core client for 30 sec - exiting 10:49:06 (44915): No heartbeat from core client for 30 sec - exiting 10:49:07 (44915): No heartbeat from core client for 30 sec - exiting 10:49:08 (44915): No heartbeat from core client for 30 sec - exiting 10:49:09 (44915): No heartbeat from core client for 30 sec - exiting 10:49:10 (44915): No heartbeat from core client for 30 sec - exiting 11:09:55 (44926): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:52:53 (59318): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:15:15 (59427): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:15:24 (59427): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(74235,0xa0969540) malloc: * error for object 0x802204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74235,0xa0969540) malloc: * error for object 0x802200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74235,0xa0969540) malloc: * error for object 0x802200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=74237, selfPID=74237, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x821e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x821e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x101f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x101f600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x1001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x1001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x1001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x1001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x1001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x1001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x821e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81897,0xa0969540) malloc: * error for object 0x821e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:11:49 (2629): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:16:37 (1477): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:52:20 (6588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:58:08 (8111): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:16:26 (331): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:30:47 (3683): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:30:48 (3683): No heartbeat from core client for 30 sec - exiting 16:30:49 (3683): No heartbeat from core client for 30 sec - exiting 00:03:27 (3807): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:34:59 (7634): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:35:00 (7634): No heartbeat from core client for 30 sec - exiting 01:35:01 (7634): No heartbeat from core client for 30 sec - exiting 03:15:44 (8405): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:15:45 (8405): No heartbeat from core client for 30 sec - exiting 03:15:46 (8405): No heartbeat from core client for 30 sec - exiting 03:15:47 (8405): No heartbeat from core client for 30 sec - exiting 03:15:48 (8405): No heartbeat from core client for 30 sec - exiting 03:15:49 (8405): No heartbeat from core client for 30 sec - exiting 08:35:29 (9313): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:30 (9313): No heartbeat from core client for 30 sec - exiting 08:35:31 (9313): No heartbeat from core client for 30 sec - exiting 08:35:32 (9313): No heartbeat from core client for 30 sec - exiting 08:35:33 (9313): No heartbeat from core client for 30 sec - exiting 08:35:34 (9313): No heartbeat from core client for 30 sec - exiting 11:40:00 (12018): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:27:45 (13669): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:27:46 (13669): No heartbeat from core client for 30 sec - exiting 14:27:47 (13669): No heartbeat from core client for 30 sec - exiting 14:27:48 (13669): No heartbeat from core client for 30 sec - exiting 14:27:49 (13669): No heartbeat from core client for 30 sec - exiting 14:27:50 (13669): No heartbeat from core client for 30 sec - exiting 14:27:51 (13669): No heartbeat from core client for 30 sec - exiting 14:27:52 (13669): No heartbeat from core client for 30 sec - exiting 14:27:53 (13669): No heartbeat from core client for 30 sec - exiting 14:27:54 (13669): No heartbeat from core client for 30 sec - exiting 14:27:55 (13669): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 16:40:42 (15118): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:24:35 (16285): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:24 (21436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:25 (21436): No heartbeat from core client for 30 sec - exiting 03:20:26 (21436): No heartbeat from core client for 30 sec - exiting 03:20:27 (21436): No heartbeat from core client for 30 sec - exiting 03:20:28 (21436): No heartbeat from core client for 30 sec - exiting 03:20:29 (21436): No heartbeat from core client for 30 sec - exiting 03:20:30 (21436): No heartbeat from core client for 30 sec - exiting 03:20:31 (21436): No heartbeat from core client for 30 sec - exiting 03:20:32 (21436): No heartbeat from core client for 30 sec - exiting 03:20:33 (21436): No heartbeat from core client for 30 sec - exiting 03:20:34 (21436): No heartbeat from core client for 30 sec - exiting 03:20:35 (21436): No heartbeat from core client for 30 sec - exiting 03:20:36 (21436): No heartbeat from core client for 30 sec - exiting 03:20:37 (21436): No heartbeat from core client for 30 sec - exiting 04:16:28 (21985): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:16:29 (21985): No heartbeat from core client for 30 sec - exiting 04:16:30 (21985): No heartbeat from core client for 30 sec - exiting 04:16:31 (21985): No heartbeat from core client for 30 sec - exiting 04:16:32 (21985): No heartbeat from core client for 30 sec - exiting 04:16:33 (21985): No heartbeat from core client for 30 sec - exiting 04:16:34 (21985): No heartbeat from core client for 30 sec - exiting 04:16:35 (21985): No heartbeat from core client for 30 sec - exiting 04:16:36 (21985): No heartbeat from core client for 30 sec - exiting 04:16:37 (21985): No heartbeat from core client for 30 sec - exiting 04:16:38 (21985): No heartbeat from core client for 30 sec - exiting 04:16:39 (21985): No heartbeat from core client for 30 sec - exiting 04:16:40 (21985): No heartbeat from core client for 30 sec - exiting 04:16:41 (21985): No heartbeat from core client for 30 sec - exiting 04:16:42 (21985): No heartbeat from core client for 30 sec - exiting 04:16:43 (21985): No heartbeat from core client for 30 sec - exiting 04:16:44 (21985): No heartbeat from core client for 30 sec - exiting 04:16:45 (21985): No heartbeat from core client for 30 sec - exiting 04:16:46 (21985): No heartbeat from core client for 30 sec - exiting 04:16:47 (21985): No heartbeat from core client for 30 sec - exiting 04:16:48 (21985): No heartbeat from core client for 30 sec - exiting 04:16:49 (21985): No heartbeat from core client for 30 sec - exiting 04:16:50 (21985): No heartbeat from core client for 30 sec - exiting 04:16:51 (21985): No heartbeat from core client for 30 sec - exiting 04:16:52 (21985): No heartbeat from core client for 30 sec - exiting 11:23:00 (22459): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:01 (22459): No heartbeat from core client for 30 sec - exiting 11:23:02 (22459): No heartbeat from core client for 30 sec - exiting 11:23:03 (22459): No heartbeat from core client for 30 sec - exiting 11:23:04 (22459): No heartbeat from core client for 30 sec - exiting 11:23:05 (22459): No heartbeat from core client for 30 sec - exiting 11:23:06 (22459): No heartbeat from core client for 30 sec - exiting 11:23:07 (22459): No heartbeat from core client for 30 sec - exiting 11:23:08 (22459): No heartbeat from core client for 30 sec - exiting 11:23:09 (22459): No heartbeat from core client for 30 sec - exiting 11:23:10 (22459): No heartbeat from core client for 30 sec - exiting 11:23:11 (22459): No heartbeat from core client for 30 sec - exiting 11:23:12 (22459): No heartbeat from core client for 30 sec - exiting 11:23:13 (22459): No heartbeat from core client for 30 sec - exiting 11:23:14 (22459): No heartbeat from core client for 30 sec - exiting 11:48:29 (26186): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:48:30 (26186): No heartbeat from core client for 30 sec - exiting 11:48:31 (26186): No heartbeat from core client for 30 sec - exiting 11:48:32 (26186): No heartbeat from core client for 30 sec - exiting 11:48:33 (26186): No heartbeat from core client for 30 sec - exiting 11:48:34 (26186): No heartbeat from core client for 30 sec - exiting 11:48:35 (26186): No heartbeat from core client for 30 sec - exiting 11:48:36 (26186): No heartbeat from core client for 30 sec - exiting 11:48:37 (26186): No heartbeat from core client for 30 sec - exiting 11:48:38 (26186): No heartbeat from core client for 30 sec - exiting 11:48:39 (26186): No heartbeat from core client for 30 sec - exiting 11:48:40 (26186): No heartbeat from core client for 30 sec - exiting 11:48:41 (26186): No heartbeat from core client for 30 sec - exiting 11:48:42 (26186): No heartbeat from core client for 30 sec - exiting 11:48:43 (26186): No heartbeat from core client for 30 sec - exiting 11:48:44 (26186): No heartbeat from core client for 30 sec - exiting 11:48:45 (26186): No heartbeat from core client for 30 sec - exiting 15:12:43 (26403): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:49:16 (31985): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:21:05 (33470): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:56:22 (33739): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:12 (34040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:05:07 (34852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:47 (36334): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:25:47 (37954): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:24:56 (41894): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:10:32 (48729): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:25:09 (51855): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:25:56 (57894): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:18 (58195): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:21:40 (58935): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:42:15 (59130): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:12:27 (59896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:13:38 (61143): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:27:55 (63704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:06:11 (70859): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:29:15 (73198): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:48:34 (75409): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:07:23 (75420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:23 (75631): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:47:27 (75741): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:08:36 (76069): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:06:38 (80839): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:51:24 (80983): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:51:58 (80983): No heartbeat from core client for 30 sec - exiting 16:09:37 (81473): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:34:09 (86665): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:13:53 (91161): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:40:00 (95136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:36:22 (97071): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:41:23 (2078): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:48:26 (8185): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:38:35 (14525): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:50:47 (14578): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:13:02 (17462): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:04:45 (2161): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:44:41 (4158): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:44:42 (4158): No heartbeat from core client for 30 sec - exiting 19:44:43 (4158): No heartbeat from core client for 30 sec - exiting 19:44:44 (4158): No heartbeat from core client for 30 sec - exiting 19:44:45 (4158): No heartbeat from core client for 30 sec - exiting 19:44:46 (4158): No heartbeat from core client for 30 sec - exiting 19:44:47 (4158): No heartbeat from core client for 30 sec - exiting 19:44:48 (4158): No heartbeat from core client for 30 sec - exiting 19:44:49 (4158): No heartbeat from core client for 30 sec - exiting 19:44:50 (4158): No heartbeat from core client for 30 sec - exiting 19:44:51 (4158): No heartbeat from core client for 30 sec - exiting 19:44:52 (4158): No heartbeat from core client for 30 sec - exiting 19:44:53 (4158): No heartbeat from core client for 30 sec - exiting 19:44:54 (4158): No heartbeat from core client for 30 sec - exiting 19:44:55 (4158): No heartbeat from core client for 30 sec - exiting 19:44:56 (4158): No heartbeat from core client for 30 sec - exiting 19:44:57 (4158): No heartbeat from core client for 30 sec - exiting 19:44:58 (4158): No heartbeat from core client for 30 sec - exiting 19:44:59 (4158): No heartbeat from core client for 30 sec - exiting 19:45:00 (4158): No heartbeat from core client for 30 sec - exiting 19:45:01 (4158): No heartbeat from core client for 30 sec - exiting 19:45:02 (4158): No heartbeat from core client for 30 sec - exiting 19:45:03 (4158): No heartbeat from core client for 30 sec - exiting 19:45:04 (4158): No heartbeat from core client for 30 sec - exiting 19:45:05 (4158): No heartbeat from core client for 30 sec - exiting 19:45:06 (4158): No heartbeat from core client for 30 sec - exiting 19:45:07 (4158): No heartbeat from core client for 30 sec - exiting 19:45:08 (4158): No heartbeat from core client for 30 sec - exiting 19:45:09 (4158): No heartbeat from core client for 30 sec - exiting 19:45:10 (4158): No heartbeat from core client for 30 sec - exiting 19:45:11 (4158): No heartbeat from core client for 30 sec - exiting 19:45:12 (4158): No heartbeat from core client for 30 sec - exiting 19:45:13 (4158): No heartbeat from core client for 30 sec - exiting 19:45:14 (4158): No heartbeat from core client for 30 sec - exiting 19:45:15 (4158): No heartbeat from core client for 30 sec - exiting 19:45:16 (4158): No heartbeat from core client for 30 sec - exiting 19:53:28 (4416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:50:34 (4490): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:50:35 (4490): No heartbeat from core client for 30 sec - exiting 20:50:36 (4490): No heartbeat from core client for 30 sec - exiting 20:50:37 (4490): No heartbeat from core client for 30 sec - exiting 20:50:38 (4490): No heartbeat from core client for 30 sec - exiting 20:50:39 (4490): No heartbeat from core client for 30 sec - exiting 20:50:40 (4490): No heartbeat from core client for 30 sec - exiting 20:50:41 (4490): No heartbeat from core client for 30 sec - exiting 20:50:42 (4490): No heartbeat from core client for 30 sec - exiting 20:50:43 (4490): No heartbeat from core client for 30 sec - exiting 20:50:44 (4490): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... 23:18:09 (6255): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:28:41 (6262): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:28:42 (6262): No heartbeat from core client for 30 sec - exiting 23:28:43 (6262): No heartbeat from core client for 30 sec - exiting 23:28:44 (6262): No heartbeat from core client for 30 sec - exiting 23:28:45 (6262): No heartbeat from core client for 30 sec - exiting 23:28:46 (6262): No heartbeat from core client for 30 sec - exiting 23:28:47 (6262): No heartbeat from core client for 30 sec - exiting 23:28:48 (6262): No heartbeat from core client for 30 sec - exiting 23:28:49 (6262): No heartbeat from core client for 30 sec - exiting 23:28:50 (6262): No heartbeat from core client for 30 sec - exiting 23:28:51 (6262): No heartbeat from core client for 30 sec - exiting 23:28:52 (6262): No heartbeat from core client for 30 sec - exiting 23:28:53 (6262): No heartbeat from core client for 30 sec - exiting 23:28:54 (6262): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 02:44:20 (8030): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:44:21 (8030): No heartbeat from core client for 30 sec - exiting 02:44:22 (8030): No heartbeat from core client for 30 sec - exiting 02:44:23 (8030): No heartbeat from core client for 30 sec - exiting 02:44:24 (8030): No heartbeat from core client for 30 sec - exiting 02:44:25 (8030): No heartbeat from core client for 30 sec - exiting 02:44:26 (8030): No heartbeat from core client for 30 sec - exiting 02:44:27 (8030): No heartbeat from core client for 30 sec - exiting 02:44:28 (8030): No heartbeat from core client for 30 sec - exiting 02:44:29 (8030): No heartbeat from core client for 30 sec - exiting 02:44:30 (8030): No heartbeat from core client for 30 sec - exiting 02:44:31 (8030): No heartbeat from core client for 30 sec - exiting 02:44:32 (8030): No heartbeat from core client for 30 sec - exiting 02:44:33 (8030): No heartbeat from core client for 30 sec - exiting 02:44:34 (8030): No heartbeat from core client for 30 sec - exiting 02:44:35 (8030): No heartbeat from core client for 30 sec - exiting 02:44:36 (8030): No heartbeat from core client for 30 sec - exiting 02:44:37 (8030): No heartbeat from core client for 30 sec - exiting 02:44:38 (8030): No heartbeat from core client for 30 sec - exiting 02:55:51 (8039): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:29:20 (10684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134390) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=696, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134390) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=696, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134390) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=696, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134390) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=696, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134390) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=696, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134390) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=696, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Oct 2012 22:14:40	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	1,036,800	1,292,195	1.2463
05 Oct 2012 11:48:07	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	1,010,880	1,259,884	1.2463
05 Oct 2012 00:05:34	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	984,960	1,226,689	1.2454
04 Oct 2012 13:14:09	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	959,040	1,193,544	1.2445
04 Oct 2012 01:36:33	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	933,120	1,160,376	1.2435
01 Oct 2012 15:21:44	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	907,200	1,127,033	1.2423
01 Oct 2012 05:04:37	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	881,280	1,093,838	1.2412
27 Sep 2012 14:55:28	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	855,360	1,060,648	1.2400
27 Sep 2012 04:13:09	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	829,440	1,027,570	1.2389
26 Sep 2012 14:00:00	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	803,520	994,404	1.2376
25 Sep 2012 23:30:30	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	777,600	961,067	1.2359
23 Sep 2012 09:36:20	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	751,680	928,428	1.2351
22 Sep 2012 22:53:42	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	725,760	897,017	1.2360
22 Sep 2012 12:38:40	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	699,840	864,340	1.2351
22 Sep 2012 02:56:44	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	673,920	833,229	1.2364
20 Sep 2012 13:12:44	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	648,000	802,970	1.2392
07 Dec 2011 07:55:51	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	622,080	773,009	1.2426
06 Dec 2011 00:57:11	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	596,160	741,027	1.2430
03 Dec 2011 20:06:17	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	570,240	707,936	1.2415
02 Dec 2011 11:47:50	1064035	13548882	hadcm3n_y9f6_1900_40_007521597_0	544,320	674,095	1.2384