Task 16608714

Name	hadcm3n_8e6r_1980_40_008727806_2
Workunit	8873784
Created	3 May 2014, 16:39:00 UTC
Sent	3 May 2014, 17:00:50 UTC
Report deadline	3 Aug 2014, 0:28:01 UTC
Received	8 Jun 2014, 8:55:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1110299
Run time	19 days 6 hours 36 min 3 sec
CPU time	18 days 5 hours 9 min 58 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.91 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:18:30 (26887): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:33 (34848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:55:56 (420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:55:59 (420): No heartbeat from core client for 30 sec - exiting SIGSEGV: segmentation violation CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x61a0004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x61bb604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x61bb600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x69aec04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x61a0004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x61bb604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x61bb600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x9a0004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x8981804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x8982604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x8982600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x5986604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x5986600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x5986604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x5986600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x498aa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x498aa00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(47477,0xad1be2c0) malloc: * error for object 0x498aa00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.5 build 11G63 Sun Jun 8 05:50:41 2014 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fab8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/28/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90021000 - 0x9004ffff /usr/lib/libSystem.B.dylib 0x900b9000 - 0x900c0fff /usr/lib/system/libsystem_dnssd.dylib 0x9165d000 - 0x91665fff /usr/lib/system/liblaunch.dylib 0x91719000 - 0x91722fff /usr/lib/libc++abi.dylib 0x91fcf000 - 0x91fd6fff /usr/lib/system/libsystem_notify.dylib 0x93924000 - 0x93924fff /usr/lib/system/libdnsinfo.dylib 0x94873000 - 0x94874fff /usr/lib/system/libunc.dylib 0x94b23000 - 0x94b39fff /usr/lib/system/libxpc.dylib 0x94fe7000 - 0x95005fff /usr/lib/system/libsystem_kernel.dylib 0x9535b000 - 0x95360fff /usr/lib/system/libmacho.dylib 0x95a7b000 - 0x95a83fff /usr/lib/system/libunwind.dylib 0x96435000 - 0x96497fff /usr/lib/libstdc++.6.dylib 0x964a4000 - 0x964d3fff /usr/lib/system/libsystem_info.dylib 0x97993000 - 0x979a1fff /usr/lib/libz.1.dylib 0x979ac000 - 0x979b0fff /usr/lib/system/libcache.dylib 0x97a0f000 - 0x97a52fff /usr/lib/system/libcommonCrypto.dylib 0x97a53000 - 0x97a56fff /usr/lib/system/libmathCommon.A.dylib 0x982e4000 - 0x982e7fff /usr/lib/system/libcompiler_rt.dylib 0x987d0000 - 0x987d8fff /usr/lib/system/libcopyfile.dylib 0x9938b000 - 0x9938cfff /usr/lib/system/libremovefile.dylib 0x9b612000 - 0x9b620fff /usr/lib/system/libdispatch.dylib 0x9c5b0000 - 0x9c67bfff /usr/lib/system/libsystem_c.dylib 0x9c763000 - 0x9c767fff /usr/lib/system/libsystem_network.dylib 0x9ca04000 - 0x9ca06fff /usr/lib/system/libdyld.dylib 0x9cdbe000 - 0x9cdbffff /usr/lib/system/libsystem_sandbox.dylib 0x9ced3000 - 0x9ced4fff /usr/lib/system/libsystem_blocks.dylib 0x9cefb000 - 0x9cefbfff /usr/lib/system/libkeymgr.dylib 0x9cf0e000 - 0x9cf0ffff /usr/lib/system/libquarantine.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Jun 2014 09:01:24	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	777,600	1,573,795	2.0239
07 Jun 2014 19:01:00	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	751,680	1,528,162	2.0330
05 Jun 2014 06:54:31	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	725,760	1,478,242	2.0368
04 Jun 2014 04:15:07	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	699,840	1,427,219	2.0394
02 Jun 2014 22:24:31	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	673,920	1,377,633	2.0442
02 Jun 2014 04:51:30	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	648,000	1,326,822	2.0476
01 Jun 2014 02:46:42	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	622,080	1,277,939	2.0543
31 May 2014 05:45:50	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	596,160	1,230,613	2.0642
29 May 2014 21:14:34	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	570,240	1,180,689	2.0705
28 May 2014 18:14:44	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	544,320	1,131,212	2.0782
27 May 2014 18:48:34	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	518,400	1,081,599	2.0864
26 May 2014 17:23:52	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	492,480	1,031,848	2.0952
25 May 2014 13:37:52	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	466,560	980,712	2.1020
24 May 2014 08:21:27	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	440,640	930,229	2.1111
23 May 2014 13:46:57	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	414,720	876,351	2.1131
22 May 2014 20:48:01	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	388,800	819,378	2.1075
21 May 2014 17:33:43	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	362,880	762,187	2.1004
20 May 2014 16:28:23	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	336,960	702,630	2.0852
18 May 2014 11:12:13	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	311,040	652,198	2.0968
17 May 2014 12:29:58	1110299	16608714	hadcm3n_8e6r_1980_40_008727806_2	285,120	599,697	2.1033