Task 19419990

Name	hadcm3n_sbln_198012_480_375_010414334_0
Workunit	10414334
Created	21 Mar 2016, 16:36:09 UTC
Sent	24 Mar 2016, 0:39:36 UTC
Report deadline	6 Mar 2017, 5:59:36 UTC
Received	17 Apr 2016, 22:43:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1377941
Run time	6 days 0 hours 36 min 28 sec
CPU time	5 days 3 hours 11 min 1 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.62 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.6.12</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:51:26 (38036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:51:27 (38036): No heartbeat from core client for 30 sec - exiting 18:51:28 (38036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:49:38 (39200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:51:32 (39215): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:08:48 (39451): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:11:33 (39464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:26:42 (43589): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:30:38 (51852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:40:51 (51999): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:42:20 (52014): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:42:21 (52014): No heartbeat from core client for 30 sec - exiting 19:42:22 (52014): No heartbeat from core client for 30 sec - exiting 19:42:23 (52014): No heartbeat from core client for 30 sec - exiting 19:42:24 (52014): No heartbeat from core client for 30 sec - exiting 19:42:25 (52014): No heartbeat from core client for 30 sec - exiting 19:42:26 (52014): No heartbeat from core client for 30 sec - exiting 19:42:27 (52014): No heartbeat from core client for 30 sec - exiting 19:42:28 (52014): No heartbeat from core client for 30 sec - exiting 19:42:29 (52014): No heartbeat from core client for 30 sec - exiting 19:42:30 (52014): No heartbeat from core client for 30 sec - exiting 19:42:31 (52014): No heartbeat from core client for 30 sec - exiting 19:42:32 (52014): No heartbeat from core client for 30 sec - exiting 19:42:33 (52014): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:10:02 (28165): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(22434,0xa3414000) malloc: * error for object 0x3815e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(22434,0xa3414000) malloc: * error for object 0x3815e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(22434,0xa3414000) malloc: * error for object 0x1809e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(24486,0xa3414000) malloc: * error for object 0x103de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24486,0xa3414000) malloc: * error for object 0x103de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24486,0xa3414000) malloc: * error for object 0x103de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(26498,0xa3414000) malloc: * error for object 0x1002004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26498,0xa3414000) malloc: * error for object 0x1002000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26498,0xa3414000) malloc: * error for object 0x1002004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(28543,0xa3414000) malloc: * error for object 0x282de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28543,0xa3414000) malloc: * error for object 0x282de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28543,0xa3414000) malloc: * error for object 0x1811c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(30579,0xa3414000) malloc: * error for object 0x282de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30579,0xa3414000) malloc: * error for object 0x282de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30579,0xa3414000) malloc: * error for object 0x282de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(32640,0xa3414000) malloc: * error for object 0x83e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32640,0xa3414000) malloc: * error for object 0x83e600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32640,0xa3414000) malloc: * error for object 0x83e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(34678,0xa3414000) malloc: * error for object 0x1825c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34678,0xa3414000) malloc: * error for object 0x1825c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34678,0xa3414000) malloc: * error for object 0x2005e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(36758,0xa3414000) malloc: * error for object 0x1800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36758,0xa3414000) malloc: * error for object 0x1800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36758,0xa3414000) malloc: * error for object 0x83e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36758,0xa3414000) malloc: * error for object 0x83e600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36758,0xa3414000) malloc: * error for object 0x83e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(38829,0xa3414000) malloc: * error for object 0x3835204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(38829,0xa3414000) malloc: * error for object 0x3835200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(38829,0xa3414000) malloc: * error for object 0x2001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.4 build 15E65 Sun Apr 17 17:43:24 2016 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fd98 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/2/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x127000 - 0x17afff /usr/lib/libstdc++.6.dylib 0x908fa000 - 0x908fafff /usr/lib/system/liblaunch.dylib 0x911c1000 - 0x911d8fff /usr/lib/system/libsystem_coretls.dylib 0x911f7000 - 0x91205fff /usr/lib/libz.1.dylib 0x91207000 - 0x9120ffff /usr/lib/system/libsystem_dnssd.dylib 0x912b1000 - 0x912b3fff /usr/lib/system/libquarantine.dylib 0x912b4000 - 0x912dcfff /usr/lib/system/libxpc.dylib 0x91365000 - 0x9138ffff /usr/lib/system/libdispatch.dylib 0x918b6000 - 0x9191dfff /usr/lib/system/libcorecrypto.dylib 0x9204f000 - 0x92050fff /usr/lib/libSystem.B.dylib 0x92054000 - 0x9207efff /usr/lib/system/libsystem_info.dylib 0x923d8000 - 0x923d8fff /usr/lib/system/libunc.dylib 0x923da000 - 0x923dffff /usr/lib/system/libcompiler_rt.dylib 0x93e11000 - 0x93e11fff /usr/lib/system/libkeymgr.dylib 0x9436b000 - 0x943d9fff /usr/lib/system/libsystem_network.dylib 0x944f2000 - 0x944f5fff /usr/lib/system/libdyld.dylib 0x94a61000 - 0x94a63fff /usr/lib/system/libsystem_secinit.dylib 0x94c1f000 - 0x94cb4fff /usr/lib/system/libsystem_c.dylib 0x94e2e000 - 0x94e2ffff /usr/lib/system/libremovefile.dylib 0x94ec5000 - 0x94edefff /usr/lib/system/libsystem_asl.dylib 0x94edf000 - 0x94f12fff /usr/lib/system/libsystem_m.dylib 0x96893000 - 0x9689bfff /usr/lib/system/libsystem_pthread.dylib 0x969b5000 - 0x969bbfff /usr/lib/system/libunwind.dylib 0x98045000 - 0x9804bfff /usr/lib/system/libsystem_platform.dylib 0x9842e000 - 0x9843ffff /usr/lib/system/libsystem_trace.dylib 0x98985000 - 0x98987fff /usr/lib/system/libsystem_configuration.dylib 0x989ff000 - 0x98a02fff /usr/lib/system/libsystem_sandbox.dylib 0x99037000 - 0x9905afff /usr/lib/libc++abi.dylib 0x9934b000 - 0x99356fff /usr/lib/system/libcommonCrypto.dylib 0x993e3000 - 0x993ecfff /usr/lib/system/libcopyfile.dylib 0x9940e000 - 0x99418fff /usr/lib/system/libsystem_notify.dylib 0x9a9d6000 - 0x9a9effff /usr/lib/system/libsystem_malloc.dylib 0x9b073000 - 0x9b073fff /usr/lib/system/libsystem_blocks.dylib 0x9b074000 - 0x9b076fff /usr/lib/system/libsystem_coreservices.dylib 0x9b433000 - 0x9b438fff /usr/lib/system/libmacho.dylib 0x9c5c2000 - 0x9c5cbfff /usr/lib/system/libsystem_networkextension.dylib 0x9d53c000 - 0x9d55dfff /usr/lib/system/libsystem_kernel.dylib 0x9d6da000 - 0x9d6defff /usr/lib/system/libcache.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Apr 2016 22:21:49	1377941	19419990	hadcm3n_sbln_198012_480_375_010414334_0	259,200	435,484	1.6801
15 Apr 2016 15:57:17	1377941	19419990	hadcm3n_sbln_198012_480_375_010414334_0	233,280	393,064	1.6849
15 Apr 2016 12:16:33	1377941	19419990	hadcm3n_sbln_198012_480_375_010414334_0	207,360	349,308	1.6845
15 Apr 2016 12:14:43	1377941	19419990	hadcm3n_sbln_198012_480_375_010414334_0	181,440	303,188	1.6710
15 Apr 2016 12:08:49	1377941	19419990	hadcm3n_sbln_198012_480_375_010414334_0	155,520	262,513	1.6880
09 Apr 2016 11:17:37	1377941	19419990	hadcm3n_sbln_198012_480_375_010414334_0	129,600	219,513	1.6938
28 Mar 2016 16:40:10	1377941	19419990	hadcm3n_sbln_198012_480_375_010414334_0	103,680	173,257	1.6711
28 Mar 2016 02:09:08	1377941	19419990	hadcm3n_sbln_198012_480_375_010414334_0	77,760	129,501	1.6654
25 Mar 2016 06:25:16	1377941	19419990	hadcm3n_sbln_198012_480_375_010414334_0	51,840	86,023	1.6594
24 Mar 2016 15:12:45	1377941	19419990	hadcm3n_sbln_198012_480_375_010414334_0	25,920	42,924	1.6560