Task 13548045

Name	hadcm3n_ya0m_1900_40_007521178_1
Workunit	7718653
Created	28 Oct 2011, 13:11:41 UTC
Sent	2 Nov 2011, 22:27:57 UTC
Report deadline	2 Feb 2012, 5:55:08 UTC
Received	21 Nov 2011, 5:42:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1159853
Run time	6 days 21 hours 29 min 11 sec
CPU time	6 days 0 hours 59 min 41 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.50 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> 19:41:58 (44772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:20:56 (502): No heartbeat from core client for 30 sec - exiting 09:20:57 (502): No heartbeat from core client for 30 sec - exiting 09:20:58 (502): No heartbeat from core client for 30 sec - exiting 09:20:59 (502): No heartbeat from core client for 30 sec - exiting 09:21:00 (502): No heartbeat from core client for 30 sec - exiting 09:21:01 (502): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:38:12 (14535): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:38:13 (14535): No heartbeat from core client for 30 sec - exiting 14:38:14 (14535): No heartbeat from core client for 30 sec - exiting 14:38:15 (14535): No heartbeat from core client for 30 sec - exiting 14:38:16 (14535): No heartbeat from core client for 30 sec - exiting 14:41:50 (35820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(42387,0xac6d62c0) malloc: * error for object 0x832b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42387,0xac6d62c0) malloc: * error for object 0xc31a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42387,0xac6d62c0) malloc: * error for object 0x5317804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42387,0xac6d62c0) malloc: * error for object 0x5317800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C74 Sun Nov 20 20:42:03 2011 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fa98 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/17/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90011000 - 0x9001afff /usr/lib/libc++abi.dylib 0x90ade000 - 0x90ae2fff /usr/lib/system/libsystem_network.dylib 0x90b63000 - 0x90b68fff /usr/lib/system/libmacho.dylib 0x91743000 - 0x91751fff /usr/lib/system/libdispatch.dylib 0x91828000 - 0x91829fff /usr/lib/system/libquarantine.dylib 0x91b72000 - 0x91bb5fff /usr/lib/system/libcommonCrypto.dylib 0x924ae000 - 0x924affff /usr/lib/system/libunc.dylib 0x92ee6000 - 0x92eedfff /usr/lib/system/libsystem_notify.dylib 0x933df000 - 0x933e7fff /usr/lib/system/liblaunch.dylib 0x9357c000 - 0x93592fff /usr/lib/system/libxpc.dylib 0x936d9000 - 0x936dbfff /usr/lib/system/libdyld.dylib 0x941bb000 - 0x941bcfff /usr/lib/system/libsystem_sandbox.dylib 0x957e1000 - 0x958acfff /usr/lib/system/libsystem_c.dylib 0x9769c000 - 0x976bafff /usr/lib/system/libsystem_kernel.dylib 0x97bd6000 - 0x97c38fff /usr/lib/libstdc++.6.dylib 0x98001000 - 0x98030fff /usr/lib/system/libsystem_info.dylib 0x990f9000 - 0x990f9fff /usr/lib/system/libdnsinfo.dylib 0x990fa000 - 0x990fbfff /usr/lib/system/libsystem_blocks.dylib 0x99417000 - 0x9941afff /usr/lib/system/libcompiler_rt.dylib 0x9941b000 - 0x9941efff /usr/lib/system/libmathCommon.A.dylib 0x9951f000 - 0x99526fff /usr/lib/system/libsystem_dnssd.dylib 0x99529000 - 0x99529fff /usr/lib/system/libkeymgr.dylib 0x99e98000 - 0x99e99fff /usr/lib/system/libremovefile.dylib 0x9b7c8000 - 0x9b7ccfff /usr/lib/system/libcache.dylib 0x9baa1000 - 0x9baaffff /usr/lib/libz.1.dylib 0x9be6b000 - 0x9be73fff /usr/lib/system/libcopyfile.dylib 0x9ca8c000 - 0x9ca94fff /usr/lib/system/libunwind.dylib 0x9cb08000 - 0x9cb36fff /usr/lib/libSystem.B.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Nov 2011 04:42:29	1159853	13548045	hadcm3n_ya0m_1900_40_007521178_1	259,200	521,974	2.0138
19 Nov 2011 22:10:15	1159853	13548045	hadcm3n_ya0m_1900_40_007521178_1	233,280	469,547	2.0128
18 Nov 2011 23:25:55	1159853	13548045	hadcm3n_ya0m_1900_40_007521178_1	207,360	417,367	2.0128
18 Nov 2011 02:52:37	1159853	13548045	hadcm3n_ya0m_1900_40_007521178_1	181,440	365,022	2.0118
17 Nov 2011 03:11:55	1159853	13548045	hadcm3n_ya0m_1900_40_007521178_1	155,520	312,874	2.0118
16 Nov 2011 09:10:11	1159853	13548045	hadcm3n_ya0m_1900_40_007521178_1	129,600	261,176	2.0152
15 Nov 2011 18:44:15	1159853	13548045	hadcm3n_ya0m_1900_40_007521178_1	103,680	209,134	2.0171
15 Nov 2011 18:44:15	1159853	13548045	hadcm3n_ya0m_1900_40_007521178_1	77,760	157,015	2.0192
15 Nov 2011 18:44:15	1159853	13548045	hadcm3n_ya0m_1900_40_007521178_1	51,840	104,908	2.0237
15 Nov 2011 18:44:15	1159853	13548045	hadcm3n_ya0m_1900_40_007521178_1	25,920	52,696	2.0330