Task 16013353

Name	hadcm3n_722z_1980_40_008413982_3
Workunit	8564838
Created	12 Sep 2013, 12:38:29 UTC
Sent	12 Sep 2013, 12:54:27 UTC
Report deadline	12 Dec 2013, 20:21:38 UTC
Received	27 Jan 2014, 20:00:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1169903
Run time	19 days 15 hours 6 min 9 sec
CPU time	15 days 21 hours 33 min 53 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.15 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:05:50 (37457): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:03:28 (38055): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:22:24 (728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:15:19 (27884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:19:17 (34017): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3837c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3837c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x483c004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x483c000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x104cc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x104cc00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x820a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x4820a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x301f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x301f600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x381c604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x701f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x5000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x5000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x7020404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x7020400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x7822004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34109,0xa05ca540) malloc: * error for object 0x7822000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug 19:48:50 (34109): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:40:27 (78928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:37:39 (35424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:19:37 (62365): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:12:39 (87883): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:37:24 (7417): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:24:09 (26021): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:59:09 (54728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:46:13 (59546): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:37:04 (92393): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:55:10 (97362): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:45 (41955): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:58:17 (68076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:32:10 (91268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:46:20 (99955): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:53:26 (81104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:30:09 (82929): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(81117,0xa05ca540) malloc: * error for object 0x2801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(583,0xa05ca540) malloc: * error for object 0x502de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(583,0xa05ca540) malloc: * error for object 0x502de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(583,0xa05ca540) malloc: * error for object 0x502de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(583,0xa05ca540) malloc: * error for object 0x502de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(583,0xa05ca540) malloc: * error for object 0x502de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(583,0xa05ca540) malloc: * error for object 0x502de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(583,0xa05ca540) malloc: * error for object 0x2001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Thu Oct 10 21:10:58 2013 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. 0 libSystem.B.dylib 0x94c9bb0f small_free_list_remove_ptr + 246 1 libSystem.B.dylib 0x94c985cc szone_free_definite_size + 3457 2 libSystem.B.dylib 0x94c975e8 free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000876a worker() + 2896 7 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 8 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 23:44:46 (583): No heartbeat from core client for 30 sec - exiting 03:49:16 (583): No heartbeat from core client for 30 sec - exiting 07:22:50 (583): No heartbeat from core client for 30 sec - exiting 02:26:41 (583): No heartbeat from core client for 30 sec - exiting 00:42:14 (583): No heartbeat from core client for 30 sec - exiting 06:26:10 (583): No heartbeat from core client for 30 sec - exiting 16:52:37 (583): No heartbeat from core client for 30 sec - exiting 15:16:22 (583): No heartbeat from core client for 30 sec - exiting 03:53:26 (583): No heartbeat from core client for 30 sec - exiting 18:33:41 (583): No heartbeat from core client for 30 sec - exiting 09:04:44 (583): No heartbeat from core client for 30 sec - exiting 01:47:15 (583): No heartbeat from core client for 30 sec - exiting 23:35:46 (583): No heartbeat from core client for 30 sec - exiting 05:18:31 (583): No heartbeat from core client for 30 sec - exiting 19:28:11 (583): No heartbeat from core client for 30 sec - exiting 02:57:31 (583): No heartbeat from core client for 30 sec - exiting 15:54:11 (583): No heartbeat from core client for 30 sec - exiting 03:21:18 (583): No heartbeat from core client for 30 sec - exiting 12:01:32 (583): No heartbeat from core client for 30 sec - exiting 00:39:45 (583): No heartbeat from core client for 30 sec - exiting 03:47:30 (583): No heartbeat from core client for 30 sec - exiting 06:31:27 (583): No heartbeat from core client for 30 sec - exiting 22:33:36 (583): No heartbeat from core client for 30 sec - exiting 06:52:41 (583): No heartbeat from core client for 30 sec - exiting 07:48:29 (583): No heartbeat from core client for 30 sec - exiting 05:21:42 (583): No heartbeat from core client for 30 sec - exiting hadcm3n_6.07_i686-apple-darwin(73610,0xa05ca540) malloc: error for object 0x8021204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(73610,0xa05ca540) malloc: * error for object 0x8021200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(24445,0xa05ca540) malloc: * error for object 0x8805c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24445,0xa05ca540) malloc: * error for object 0x8805c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24445,0xa05ca540) malloc: * error for object 0x1021e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Jan 27 13:57:56 2014 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. 0 libSystem.B.dylib 0x94c9bb0f small_free_list_remove_ptr + 246 1 libSystem.B.dylib 0x94c985cc szone_free_definite_size + 3457 2 libSystem.B.dylib 0x94c975e8 free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000876a worker() + 2896 7 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 8 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x94c90c0e mach_wait_until + 10 1 libSystem.B.dylib 0x94d18429 nanosleep + 345 2 libSystem.B.dylib 0x94d182ca usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c boinc_sleep(double) + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 timer_thread(void) + 78 5 libSystem.B.dylib 0x94cbe259 _pthread_start + 345 6 libSystem.B.dylib 0x94cbe0de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f9b8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/16/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x93700000 - 0x9370efff /usr/lib/libz.1.dylib 0x94c90000 - 0x94e37fff /usr/lib/libSystem.B.dylib 0x94e74000 - 0x94e77fff /usr/lib/system/libmathCommon.A.dylib 0x97fcf000 - 0x98039fff /usr/lib/libstdc++.6.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Oct 2013 00:05:20	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	777,600	1,373,026	1.7657
10 Oct 2013 08:15:47	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	751,680	1,325,791	1.7638
09 Oct 2013 16:09:56	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	725,760	1,278,385	1.7614
09 Oct 2013 00:48:58	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	699,840	1,231,175	1.7592
08 Oct 2013 07:33:25	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	673,920	1,184,062	1.7570
06 Oct 2013 20:52:08	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	648,000	1,136,866	1.7544
06 Oct 2013 02:36:15	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	622,080	1,090,183	1.7525
05 Oct 2013 07:03:21	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	596,160	1,043,371	1.7502
04 Oct 2013 12:59:01	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	570,240	996,273	1.7471
03 Oct 2013 18:33:28	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	544,320	949,806	1.7449
03 Oct 2013 00:07:00	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	518,400	902,874	1.7417
02 Oct 2013 05:02:18	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	492,480	855,958	1.7381
01 Oct 2013 10:01:27	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	466,560	809,150	1.7343
30 Sep 2013 15:29:30	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	440,640	762,018	1.7293
29 Sep 2013 20:24:35	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	414,720	714,515	1.7229
29 Sep 2013 02:21:41	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	388,800	666,866	1.7152
28 Sep 2013 12:11:05	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	362,880	619,443	1.7070
27 Sep 2013 13:44:37	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	336,960	572,426	1.6988
26 Sep 2013 22:49:10	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	311,040	525,729	1.6902
26 Sep 2013 06:31:14	1169903	16013353	hadcm3n_722z_1980_40_008413982_3	285,120	478,619	1.6787