Task 16081871

Name	hadcm3n_o2zy_1940_40_008404572_4
Workunit	8555428
Created	18 Nov 2013, 20:06:30 UTC
Sent	18 Nov 2013, 20:06:36 UTC
Report deadline	18 Feb 2014, 3:33:47 UTC
Received	17 Jan 2014, 11:01:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1001397
Run time	20 days 9 hours 33 min 18 sec
CPU time	19 days 6 hours 36 min 13 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(42391,0xa0c87540) malloc: * error for object 0x4801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42391,0xa0c87540) malloc: * error for object 0x4801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42391,0xa0c87540) malloc: * error for object 0x2004e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=42395, selfPID=42395, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(82393,0xa0c87540) malloc: * error for object 0x4019c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(82393,0xa0c87540) malloc: * error for object 0x4019c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(82393,0xa0c87540) malloc: * error for object 0x4019c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(82393,0xa0c87540) malloc: * error for object 0x4019c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(82393,0xa0c87540) malloc: * error for object 0x4019c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(22313,0xa0c87540) malloc: * error for object 0x382a604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(22313,0xa0c87540) malloc: * error for object 0x382a600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(22313,0xa0c87540) malloc: * error for object 0x382a604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(81361,0xa0c87540) malloc: * error for object 0x3800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81361,0xa0c87540) malloc: * error for object 0x3800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81361,0xa0c87540) malloc: * error for object 0x3800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81361,0xa0c87540) malloc: * error for object 0x3800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81361,0xa0c87540) malloc: * error for object 0x3800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81361,0xa0c87540) malloc: * error for object 0x3800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81361,0xa0c87540) malloc: * error for object 0x3800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(22042,0xa0c87540) malloc: * error for object 0x280e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Wed Jan 15 22:00:10 2014 hadcm3n_6.07_i686-apple-darwin(17721,0xa0c87540) malloc: * error for object 0x1004e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(17721,0xa0c87540) malloc: * error for object 0x1005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(17721,0xa0c87540) malloc: * error for object 0x1005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(17721,0xa0c87540) malloc: * error for object 0x1005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Fri Jan 17 22:00:12 2014 Thread 0 Crashed: 0 libSystem.B.dylib 0x9a4c4b03 _small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x9a4c15cc _szone_free_definite_size + 3457 2 libSystem.B.dylib 0x9a4c05e8 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000876a __Z6workerv + 2896 7 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 8 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x9a4b9c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x9a541429 _nanosleep + 345 2 libSystem.B.dylib 0x9a5412ca _usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c __Z11boinc_sleepd + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9a4e7259 __pthread_start + 345 6 libSystem.B.dylib 0x9a4e70de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fb38 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/4/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x9173a000 - 0x917a4fff /usr/lib/libstdc++.6.dylib 0x940e8000 - 0x940f6fff /usr/lib/libz.1.dylib 0x9a4b9000 - 0x9a660fff /usr/lib/libSystem.B.dylib 0x9abda000 - 0x9abddfff /usr/lib/system/libmathCommon.A.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Jan 2014 15:28:08	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	1,036,800	1,664,634	1.6055
13 Jan 2014 13:05:20	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	1,010,880	1,621,172	1.6037
11 Jan 2014 17:08:55	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	984,960	1,577,706	1.6018
10 Jan 2014 17:05:30	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	959,040	1,534,135	1.5997
08 Jan 2014 18:07:19	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	933,120	1,490,636	1.5975
07 Jan 2014 15:05:48	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	907,200	1,447,929	1.5960
06 Jan 2014 13:06:18	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	881,280	1,409,244	1.5991
04 Jan 2014 19:06:29	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	855,360	1,370,587	1.6024
03 Jan 2014 18:05:25	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	829,440	1,333,534	1.6078
02 Jan 2014 17:06:04	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	803,520	1,299,258	1.6170
01 Jan 2014 16:13:57	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	777,600	1,264,784	1.6265
31 Dec 2013 13:00:48	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	751,680	1,222,186	1.6259
29 Dec 2013 18:08:10	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	725,760	1,180,442	1.6265
28 Dec 2013 14:01:54	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	699,840	1,135,389	1.6224
26 Dec 2013 20:08:46	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	673,920	1,091,600	1.6198
25 Dec 2013 16:12:23	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	648,000	1,047,787	1.6170
24 Dec 2013 12:04:22	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	622,080	1,003,882	1.6138
22 Dec 2013 17:21:12	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	596,160	959,941	1.6102
21 Dec 2013 13:41:23	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	570,240	916,125	1.6066
19 Dec 2013 19:04:22	1001397	16081871	hadcm3n_o2zy_1940_40_008404572_4	544,320	872,206	1.6024