Task 16291494

Name	hadcm3n_n09h_1960_40_008399294_4
Workunit	8550150
Created	18 Feb 2014, 12:00:16 UTC
Sent	18 Feb 2014, 12:00:51 UTC
Report deadline	20 May 2014, 19:28:02 UTC
Received	6 Apr 2014, 12:01:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1001397
Run time	16 days 1 hours 16 min 38 sec
CPU time	15 days 8 hours 11 min 14 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(19413,0xa0c87540) malloc: * error for object 0x2000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Wed Mar 19 02:41:34 2014 Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=19419, selfPID=19419, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(55792,0xa0c87540) malloc: * error for object 0x4804e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(55792,0xa0c87540) malloc: * error for object 0x4804e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(55792,0xa0c87540) malloc: * error for object 0x4804e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(55792,0xa0c87540) malloc: * error for object 0x4804e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(55792,0xa0c87540) malloc: * error for object 0x3005e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(55792,0xa0c87540) malloc: * error for object 0x3005e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(55792,0xa0c87540) malloc: * error for object 0x3001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 SIGSEGV: segmentation violation CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x4840004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x4840000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x2036204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x4054604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x4054600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x283a004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x381fa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x381fa00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x301fa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x301fa00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x1045804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x1045800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x2036204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x2036200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x1012204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x1012200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x3001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x3001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x2805e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x2805e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2371,0xa0c87540) malloc: * error for object 0x2805e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2376, selfPID=2376, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(38777,0xa0c87540) malloc: * error for object 0x84b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(38777,0xa0c87540) malloc: * error for object 0x84b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x2844604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x401b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x2844604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x401b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x103a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x103a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x103a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x103a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x103a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x103a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x103a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x103a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(75094,0xa0c87540) malloc: * error for object 0x103a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(13086,0xa0c87540) malloc: * error for object 0x1019e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sun Apr 6 22:00:06 2014 Thread 0 Crashed: 0 libSystem.B.dylib 0x9a4c4b0f _small_free_list_remove_ptr + 246 1 libSystem.B.dylib 0x9a4c15cc _szone_free_definite_size + 3457 2 libSystem.B.dylib 0x9a4c05e8 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000876a __Z6workerv + 2896 7 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 8 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x9a4b9c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x9a541429 _nanosleep + 345 2 libSystem.B.dylib 0x9a5412ca _usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c __Z11boinc_sleepd + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9a4e7259 __pthread_start + 345 6 libSystem.B.dylib 0x9a4e70de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fb18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/5/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x9173a000 - 0x917a4fff /usr/lib/libstdc++.6.dylib 0x940e8000 - 0x940f6fff /usr/lib/libz.1.dylib 0x9a4b9000 - 0x9a660fff /usr/lib/libSystem.B.dylib 0x9abda000 - 0x9abddfff /usr/lib/system/libmathCommon.A.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Apr 2014 16:25:45	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	777,600	1,309,917	1.6846
02 Apr 2014 12:46:52	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	751,680	1,266,387	1.6847
01 Apr 2014 01:13:10	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	725,760	1,220,382	1.6815
30 Mar 2014 13:38:44	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	699,840	1,176,175	1.6806
28 Mar 2014 18:37:50	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	673,920	1,131,905	1.6796
27 Mar 2014 14:55:12	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	648,000	1,087,636	1.6785
26 Mar 2014 11:15:30	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	622,080	1,043,344	1.6772
24 Mar 2014 18:24:08	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	596,160	1,003,405	1.6831
23 Mar 2014 14:39:46	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	570,240	960,653	1.6846
21 Mar 2014 19:42:35	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	544,320	916,524	1.6838
18 Mar 2014 22:55:22	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	518,400	871,483	1.6811
17 Mar 2014 13:06:02	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	492,480	827,652	1.6806
15 Mar 2014 17:33:21	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	466,560	783,767	1.6799
14 Mar 2014 14:00:27	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	440,640	739,896	1.6791
12 Mar 2014 20:10:00	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	414,720	695,936	1.6781
11 Mar 2014 20:39:14	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	388,800	652,194	1.6775
10 Mar 2014 11:56:34	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	362,880	608,258	1.6762
09 Mar 2014 03:09:07	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	336,960	564,486	1.6752
07 Mar 2014 13:45:26	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	311,040	520,738	1.6742
05 Mar 2014 19:08:07	1001397	16291494	hadcm3n_n09h_1960_40_008399294_4	285,120	476,834	1.6724