Task 15904103

Name	hadcm3n_38b7_2020_40_008404153_0
Workunit	8555009
Created	23 Jul 2013, 17:49:33 UTC
Sent	23 Jul 2013, 20:21:58 UTC
Report deadline	23 Oct 2013, 3:49:09 UTC
Received	14 Aug 2013, 16:04:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	987869
Run time	10 days 16 hours 43 min 23 sec
CPU time	10 days 1 hours 10 min 44 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.75 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(5011,0xa0805540) malloc: * error for object 0x1035604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5011,0xa0805540) malloc: * error for object 0x1050c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5011,0xa0805540) malloc: * error for object 0x1050c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5011,0xa0805540) malloc: * error for object 0x2041204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5011,0xa0805540) malloc: * error for object 0x4020a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5011,0xa0805540) malloc: * error for object 0x2041204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x850e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x401f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x850e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x2021404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x2021400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x2005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x2005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x2005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x2005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x2005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x2005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x3001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x3001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x3001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28348,0xa0805540) malloc: * error for object 0x3001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:12:21 (74376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:24:09 (154): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:19:40 (10517): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:45:45 (12398): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:27:18 (13235): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:54:39 (22732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:40:53 (27631): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:07:46 (29326): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:25:44 (30221): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:56:13 (38883): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:19:03 (39893): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:36:42 (40656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:05:34 (41242): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:25:24 (42215): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:49:49 (42867): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:14:39 (43685): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:44:55 (44509): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:17:38 (45527): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:45:33 (46608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:16:09 (47547): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:38:22 (48559): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:03:37 (49300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:37:00 (50146): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:57:18 (51255): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:25:07 (51934): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:46:42 (52857): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:07:15 (53576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:31:35 (54259): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:49:23 (55069): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:16:47 (55671): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:38:10 (56577): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:03:05 (57291): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:05 (58129): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:53:49 (59059): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:13:55 (59813): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:41:39 (60481): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:06:09 (61407): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:35:54 (62226): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:36 (63212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:23:50 (64073): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:52:20 (64806): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:11:04 (65764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:36:20 (66380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:54:24 (67227): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:15:18 (67820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:36:42 (68525): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:55:21 (69235): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:18:39 (69855): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:51:12 (70627): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:21:17 (71723): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:50:33 (72715): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:09:45 (73697): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:34:18 (74337): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:54:25 (75162): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:22:26 (75821): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:46:00 (76836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:15:41 (77621): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... hadcm3n_6.07_i686-apple-darwin(78610,0xa0805540) malloc: * error for object 0x301b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78610,0xa0805540) malloc: * error for object 0x301b600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Tue Aug 13 08:37:19 2013 hadcm3n_6.07_i686-apple-darwin(78610,0xa0805540) malloc: * error for object 0x282fa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78610,0xa0805540) malloc: * error for object 0x2830204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78610,0xa0805540) malloc: * error for object 0x2830a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78610,0xa0805540) malloc: * error for object 0x2830e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78610,0xa0805540) malloc: * error for object 0x2831404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78610,0xa0805540) malloc: * error for object 0x2831e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78610,0xa0805540) malloc: * error for object 0x2831e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin. 0 libSystem.B.dylib 0x96fded3a small_malloc_from_free_list + 762 1 libSystem.B.dylib 0x96fdb4d9 szone_malloc_should_clear + 2271 2 libSystem.B.dylib 0x96fdaba8 malloc_zone_malloc + 81 3 libSystem.B.dylib 0x96fd8c78 malloc + 50 4 libstdc++.6.dylib 0x9254c617 operator new(unsigned long) + 36 5 libstdc++.6.dylib 0x9254c703 operator new[](unsigned long) + 17 6 hadcm3n_6.07_i686-apple-darwin 0x0000b8e7 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3113 7 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 8 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 9 hadcm3n_6.07_i686-apple-darwin 0x0000791c mainLoop() + 410 10 hadcm3n_6.07_i686-apple-darwin 0x000087c7 worker() + 2989 11 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 12 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x96fd6c0e mach_wait_until + 10 1 libSystem.B.dylib 0x9705e429 nanosleep + 345 2 libSystem.B.dylib 0x9705e2ca usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c boinc_sleep(double) + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 timer_thread(void) + 78 5 libSystem.B.dylib 0x97004259 _pthread_start + 345 6 libSystem.B.dylib 0x970040de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f668 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/7/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x91739000 - 0x9173cfff /usr/lib/system/libmathCommon.A.dylib 0x92502000 - 0x9256cfff /usr/lib/libstdc++.6.dylib 0x96fd6000 - 0x9717dfff /usr/lib/libSystem.B.dylib 0x97347000 - 0x97355fff /usr/lib/libz.1.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	518,400	868,239	1.6748
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	492,480	825,997	1.6772
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	466,560	786,124	1.6849
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	440,640	746,436	1.6940
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	414,720	706,978	1.7047
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	388,800	669,014	1.7207
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	362,880	632,225	1.7422
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	336,960	595,037	1.7659
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	311,040	557,230	1.7915
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	285,120	511,838	1.7952
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	259,200	465,956	1.7977
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	233,280	419,009	1.7962
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	207,360	374,086	1.8040
14 Aug 2013 16:12:06	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	181,440	328,298	1.8094
14 Aug 2013 16:12:05	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	155,520	282,191	1.8145
14 Aug 2013 16:12:05	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	129,600	234,872	1.8123
14 Aug 2013 16:12:05	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	103,680	188,850	1.8215
30 Jul 2013 09:46:39	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	77,760	143,211	1.8417
26 Jul 2013 00:44:15	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	51,840	95,559	1.8433
24 Jul 2013 19:10:50	987869	15904103	hadcm3n_38b7_2020_40_008404153_0	25,920	49,012	1.8909