Task 13383552

Name	hadcm3n_p57t_1940_40_007450590_2
Workunit	7648093
Created	13 Sep 2011, 12:44:08 UTC
Sent	13 Sep 2011, 12:45:32 UTC
Report deadline	13 Dec 2011, 20:12:43 UTC
Received	15 Oct 2011, 16:24:17 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1291115
Run time	18 days 2 hours 42 min 6 sec
CPU time	16 days 20 hours 45 min 39 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.91 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.41</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 15:43:43 (53661): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:43:44 (53661): No heartbeat from core client for 30 sec - exiting 15:43:45 (53661): No heartbeat from core client for 30 sec - exiting 15:43:46 (53661): No heartbeat from core client for 30 sec - exiting 15:43:47 (53661): No heartbeat from core client for 30 sec - exiting 15:43:48 (53661): No heartbeat from core client for 30 sec - exiting 15:43:49 (53661): No heartbeat from core client for 30 sec - exiting 15:43:50 (53661): No heartbeat from core client for 30 sec - exiting 15:43:51 (53661): No heartbeat from core client for 30 sec - exiting 15:43:52 (53661): No heartbeat from core client for 30 sec - exiting 15:43:53 (53661): No heartbeat from core client for 30 sec - exiting 15:43:54 (53661): No heartbeat from core client for 30 sec - exiting 15:58:57 (53779): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:58:58 (53779): No heartbeat from core client for 30 sec - exiting 15:58:59 (53779): No heartbeat from core client for 30 sec - exiting 15:59:00 (53779): No heartbeat from core client for 30 sec - exiting 15:59:01 (53779): No heartbeat from core client for 30 sec - exiting 15:59:02 (53779): No heartbeat from core client for 30 sec - exiting 15:59:03 (53779): No heartbeat from core client for 30 sec - exiting 15:59:04 (53779): No heartbeat from core client for 30 sec - exiting 15:59:05 (53779): No heartbeat from core client for 30 sec - exiting 15:59:06 (53779): No heartbeat from core client for 30 sec - exiting 15:59:07 (53779): No heartbeat from core client for 30 sec - exiting 15:59:08 (53779): No heartbeat from core client for 30 sec - exiting 15:59:09 (53779): No heartbeat from core client for 30 sec - exiting 15:59:10 (53779): No heartbeat from core client for 30 sec - exiting 15:59:11 (53779): No heartbeat from core client for 30 sec - exiting 15:59:12 (53779): No heartbeat from core client for 30 sec - exiting 15:59:13 (53779): No heartbeat from core client for 30 sec - exiting 15:59:14 (53779): No heartbeat from core client for 30 sec - exiting 15:59:15 (53779): No heartbeat from core client for 30 sec - exiting 15:59:16 (53779): No heartbeat from core client for 30 sec - exiting 15:59:17 (53779): No heartbeat from core client for 30 sec - exiting 15:59:18 (53779): No heartbeat from core client for 30 sec - exiting 15:59:19 (53779): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:01:44 (75935): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:59:04 (57255): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:33:46 (67306): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:33:48 (67306): No heartbeat from core client for 30 sec - exiting 06:13:24 (74412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:33:59 (82298): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:16:07 (87947): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:17:27 (90398): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:17:29 (90398): No heartbeat from core client for 30 sec - exiting 10:17:30 (90398): No heartbeat from core client for 30 sec - exiting 14:14:33 (97295): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:14:34 (97295): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:01:30 (17565): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:25:59 (27389): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:25:16 (4640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:53:34 (43739): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:34:29 (49935): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:30:29 (47458): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:41:34 (61880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:53:18 (63955): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:20:14 (66227): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:24:17 (66936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:24:19 (66936): No heartbeat from core client for 30 sec - exiting 10:24:20 (66936): No heartbeat from core client for 30 sec - exiting 10:24:21 (66936): No heartbeat from core client for 30 sec - exiting 10:24:22 (66936): No heartbeat from core client for 30 sec - exiting 10:24:23 (66936): No heartbeat from core client for 30 sec - exiting 10:24:24 (66936): No heartbeat from core client for 30 sec - exiting 10:24:25 (66936): No heartbeat from core client for 30 sec - exiting 10:24:27 (66936): No heartbeat from core client for 30 sec - exiting 10:24:28 (66936): No heartbeat from core client for 30 sec - exiting 10:24:29 (66936): No heartbeat from core client for 30 sec - exiting 10:24:30 (66936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:20:24 (68846): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:20:25 (68846): No heartbeat from core client for 30 sec - exiting 12:20:26 (68846): No heartbeat from core client for 30 sec - exiting 12:20:27 (68846): No heartbeat from core client for 30 sec - exiting 12:20:28 (68846): No heartbeat from core client for 30 sec - exiting 12:20:29 (68846): No heartbeat from core client for 30 sec - exiting 12:20:30 (68846): No heartbeat from core client for 30 sec - exiting 12:21:51 (69695): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:21:52 (69695): No heartbeat from core client for 30 sec - exiting 12:21:53 (69695): No heartbeat from core client for 30 sec - exiting 12:21:54 (69695): No heartbeat from core client for 30 sec - exiting 12:21:55 (69695): No heartbeat from core client for 30 sec - exiting 12:21:56 (69695): No heartbeat from core client for 30 sec - exiting 12:21:57 (69695): No heartbeat from core client for 30 sec - exiting 12:21:58 (69695): No heartbeat from core client for 30 sec - exiting 12:21:59 (69695): No heartbeat from core client for 30 sec - exiting 12:22:00 (69695): No heartbeat from core client for 30 sec - exiting 12:22:01 (69695): No heartbeat from core client for 30 sec - exiting 12:22:02 (69695): No heartbeat from core client for 30 sec - exiting 12:22:03 (69695): No heartbeat from core client for 30 sec - exiting 12:22:04 (69695): No heartbeat from core client for 30 sec - exiting 12:22:05 (69695): No heartbeat from core client for 30 sec - exiting 12:22:06 (69695): No heartbeat from core client for 30 sec - exiting 12:22:07 (69695): No heartbeat from core client for 30 sec - exiting 12:22:08 (69695): No heartbeat from core client for 30 sec - exiting 12:22:09 (69695): No heartbeat from core client for 30 sec - exiting 12:22:10 (69695): No heartbeat from core client for 30 sec - exiting 12:22:11 (69695): No heartbeat from core client for 30 sec - exiting 12:22:12 (69695): No heartbeat from core client for 30 sec - exiting 12:22:13 (69695): No heartbeat from core client for 30 sec - exiting 12:22:14 (69695): No heartbeat from core client for 30 sec - exiting 12:22:15 (69695): No heartbeat from core client for 30 sec - exiting 12:22:16 (69695): No heartbeat from core client for 30 sec - exiting 12:22:17 (69695): No heartbeat from core client for 30 sec - exiting 12:22:18 (69695): No heartbeat from core client for 30 sec - exiting 12:22:19 (69695): No heartbeat from core client for 30 sec - exiting 12:22:20 (69695): No heartbeat from core client for 30 sec - exiting 12:22:21 (69695): No heartbeat from core client for 30 sec - exiting 12:22:22 (69695): No heartbeat from core client for 30 sec - exiting 12:22:23 (69695): No heartbeat from core client for 30 sec - exiting 12:22:24 (69695): No heartbeat from core client for 30 sec - exiting 12:22:25 (69695): No heartbeat from core client for 30 sec - exiting 12:22:26 (69695): No heartbeat from core client for 30 sec - exiting 12:22:27 (69695): No heartbeat from core client for 30 sec - exiting 12:22:28 (69695): No heartbeat from core client for 30 sec - exiting 12:22:29 (69695): No heartbeat from core client for 30 sec - exiting 12:22:30 (69695): No heartbeat from core client for 30 sec - exiting 12:56:10 (69746): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:01:11 (71768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:11:13 (71840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:54:10 (72353): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:23:20 (60758): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:59:15 (8011): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:59:16 (8011): No heartbeat from core client for 30 sec - exiting 06:10:57 (9319): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:10:58 (9319): No heartbeat from core client for 30 sec - exiting 06:15:57 (23730): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:15:59 (23730): No heartbeat from core client for 30 sec - exiting 06:16:00 (23730): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:19:05 (39342): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:54:56 (39562): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:57:20 (42908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:57:21 (42908): No heartbeat from core client for 30 sec - exiting 10:57:22 (42908): No heartbeat from core client for 30 sec - exiting 10:57:23 (42908): No heartbeat from core client for 30 sec - exiting 10:57:24 (42908): No heartbeat from core client for 30 sec - exiting 10:57:25 (42908): No heartbeat from core client for 30 sec - exiting 10:57:26 (42908): No heartbeat from core client for 30 sec - exiting 10:57:27 (42908): No heartbeat from core client for 30 sec - exiting 10:57:28 (42908): No heartbeat from core client for 30 sec - exiting 10:57:29 (42908): No heartbeat from core client for 30 sec - exiting 10:57:30 (42908): No heartbeat from core client for 30 sec - exiting 10:57:31 (42908): No heartbeat from core client for 30 sec - exiting 10:57:32 (42908): No heartbeat from core client for 30 sec - exiting 10:57:33 (42908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:55:31 (78320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:07:12 (89436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:12:43 (11058): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:17:02 (11194): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:39:30 (18892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:13:40 (22392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:13:42 (22392): No heartbeat from core client for 30 sec - exiting 05:13:43 (22392): No heartbeat from core client for 30 sec - exiting 05:25:53 (24394): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:25:55 (24394): No heartbeat from core client for 30 sec - exiting 05:33:43 (27224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:53:44 (29926): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:52 (30398): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:53 (30398): No heartbeat from core client for 30 sec - exiting 05:55:54 (30398): No heartbeat from core client for 30 sec - exiting 05:55:55 (30398): No heartbeat from core client for 30 sec - exiting 05:58:07 (31728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:01:07 (33078): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:03:48 (33208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:07:41 (33301): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:07:42 (33301): No heartbeat from core client for 30 sec - exiting 06:13:50 (33419): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:13:52 (33419): No heartbeat from core client for 30 sec - exiting 06:13:53 (33419): No heartbeat from core client for 30 sec - exiting 06:13:54 (33419): No heartbeat from core client for 30 sec - exiting 06:18:51 (33600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:52 (33600): No heartbeat from core client for 30 sec - exiting 06:21:26 (33690): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:21:28 (33690): No heartbeat from core client for 30 sec - exiting 06:21:29 (33690): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0xa824604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0xb81f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0xb81f600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0x6836204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0x1035204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0x782ee04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0x1034604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0x1034600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0x4001204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0x4002004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0x4002000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48912,0xa0aa2540) malloc: * error for object 0x4002004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Sep 26 11:54:02 2011 Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=48913, selfPID=48913, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x681b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x481b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x681b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x7808e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(495,0xa0aa2540) malloc: * error for object 0x7808e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug 15:18:43 (495): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:18:45 (495): No heartbeat from core client for 30 sec - exiting 15:18:46 (495): No heartbeat from core client for 30 sec - exiting 15:21:31 (6486): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:10:07 (80734): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:10:02 (16518): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:33:20 (93553): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:56:53 (17747): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:35 (21088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:12:50 (21202): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:57:33 (61131): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:57:34 (61131): No heartbeat from core client for 30 sec - exiting 15:57:35 (61131): No heartbeat from core client for 30 sec - exiting 15:57:36 (61131): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:09:59 (42861): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:20:09 (66176): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:06:11 (23986): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(324,0xa04fe540) malloc: * error for object 0x9004e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sat Oct 15 11:22:35 2011 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin. 0 libSystem.B.dylib 0x93b41b03 small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x93b3e546 szone_free_definite_size + 3323 2 libSystem.B.dylib 0x93b3d5e8 free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000b99a annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3292 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000791c mainLoop() + 410 7 hadcm3n_6.07_i686-apple-darwin 0x000087c7 worker() + 2989 8 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 9 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x93b36c0e mach_wait_until + 10 1 libSystem.B.dylib 0x93bbe429 nanosleep + 345 2 libSystem.B.dylib 0x93bbe2ca usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c boinc_sleep(double) + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 timer_thread(void) + 78 5 libSystem.B.dylib 0x93b64259 _pthread_start + 345 6 libSystem.B.dylib 0x93b640de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f858 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/21/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x9026c000 - 0x9027afff /usr/lib/libz.1.dylib 0x92415000 - 0x92418fff /usr/lib/system/libmathCommon.A.dylib 0x93b36000 - 0x93cddfff /usr/lib/libSystem.B.dylib 0x9a630000 - 0x9a69afff /usr/lib/libstdc++.6.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Oct 2011 16:29:21	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	777,600	1,457,131	1.8739
15 Oct 2011 02:42:07	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	751,680	1,410,732	1.8768
14 Oct 2011 12:17:23	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	725,760	1,363,979	1.8794
13 Oct 2011 21:08:46	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	699,840	1,317,186	1.8821
13 Oct 2011 06:21:52	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	673,920	1,271,306	1.8864
12 Oct 2011 14:25:22	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	648,000	1,225,230	1.8908
11 Oct 2011 23:26:50	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	622,080	1,176,696	1.8916
09 Oct 2011 22:09:13	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	596,160	1,128,106	1.8923
08 Oct 2011 12:59:03	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	570,240	1,075,314	1.8857
07 Oct 2011 21:11:38	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	544,320	1,022,019	1.8776
07 Oct 2011 00:16:03	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	518,400	973,751	1.8784
06 Oct 2011 07:37:08	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	492,480	922,815	1.8738
05 Oct 2011 16:49:00	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	466,560	873,110	1.8714
05 Oct 2011 14:36:02	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	440,640	825,270	1.8729
05 Oct 2011 14:36:02	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	414,720	776,995	1.8735
05 Oct 2011 14:36:02	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	388,800	730,915	1.8799
05 Oct 2011 14:36:02	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	362,880	682,930	1.8820
05 Oct 2011 14:36:02	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	336,960	634,182	1.8821
05 Oct 2011 14:36:02	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	311,040	586,312	1.8850
05 Oct 2011 14:36:02	1099001	13383552	hadcm3n_p57t_1940_40_007450590_2	285,120	538,086	1.8872