Task 12749213

Name	hadcm3n_o6oq_1900_40_007203997_1
Workunit	7402277
Created	28 Mar 2011, 14:17:50 UTC
Sent	29 Mar 2011, 0:09:51 UTC
Report deadline	28 Jun 2011, 7:37:02 UTC
Received	10 May 2011, 7:15:19 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1136193
Run time	21 days 23 hours 21 min 53 sec
CPU time	14 days 22 hours 31 min 4 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	3.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Wed Mar 30 00:23:57 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x0041be30 _expf.J + 192 1 hadcm3n_um_6.07_i686-apple-darwin 0x0002ce1c _fast_tanh. + 255 2 hadcm3n_um_6.07_i686-apple-darwin 0x0002be15 _ai_calc + 32325 3 hadcm3n_um_6.07_i686-apple-darwin 0x001a2fc4 _isopyc_m + 4068 4 hadcm3n_um_6.07_i686-apple-darwin 0x002fe3b9 _tracer + 14633 5 hadcm3n_um_6.07_i686-apple-darwin 0x0027b525 _rowcalc + 38533 6 hadcm3n_um_6.07_i686-apple-darwin 0x00056cf6 _blokcntl + 12566 7 hadcm3n_um_6.07_i686-apple-darwin 0x0005012b _blokcalc + 16843 8 hadcm3n_um_6.07_i686-apple-darwin 0x00287e62 _row_ctl + 9426 9 hadcm3n_um_6.07_i686-apple-darwin 0x001f9a84 _ocn_ctl + 7252 10 hadcm3n_um_6.07_i686-apple-darwin 0x001f7253 _ocn_for_step + 6755 11 hadcm3n_um_6.07_i686-apple-darwin 0x00380efd _ocn_step + 3693 12 hadcm3n_um_6.07_i686-apple-darwin 0x0039539d _u_model + 45245 13 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 14 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 15 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 16 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbddb7358 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=63549, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Wed Mar 30 02:25:46 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x0026d8a4 _relax + 16484 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032b2fb _trop_ctl + 4891 2 hadcm3n_um_6.07_i686-apple-darwin 0x001fa6de _ocn_ctl + 10414 3 hadcm3n_um_6.07_i686-apple-darwin 0x001f7253 _ocn_for_step + 6755 4 hadcm3n_um_6.07_i686-apple-darwin 0x00380efd _ocn_step + 3693 5 hadcm3n_um_6.07_i686-apple-darwin 0x0039539d _u_model + 45245 6 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 7 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 8 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 9 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbe8ce4c8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=63549, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(556,0xa06f8540) malloc: * error for object 0x802c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(556,0xa06f8540) malloc: * error for object 0x802c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(556,0xa06f8540) malloc: * error for object 0x2801204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(556,0xa06f8540) malloc: * error for object 0x2802004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(556,0xa06f8540) malloc: * error for object 0x2802000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(556,0xa06f8540) malloc: * error for object 0x2802004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(556,0xa06f8540) malloc: * error for object 0x2802000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(556,0xa06f8540) malloc: * error for object 0x2802004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(556,0xa06f8540) malloc: * error for object 0x2802000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(556,0xa06f8540) malloc: * error for object 0x800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(556,0xa06f8540) malloc: * error for object 0x800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x3032a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x831004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x831004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x84c604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x84c600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x3032a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x2827404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x2827400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x2820c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x2820c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x2820c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x2820c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x2820c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x2820c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x2820c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x2820c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x1020c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x1021a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x1021a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x1024a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x1024a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40903,0xa00a1540) malloc: * error for object 0x201f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Sat Apr 30 05:38:41 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x002bb9e0 _solver_triple + 24928 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032d6ef _triple_column + 2479 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x92fa41ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9302b809 _nanosleep + 345 2 libSystem.B.dylib 0x9302b6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x92fd17fd __pthread_start + 345 6 libSystem.B.dylib 0x92fd1682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbda51ba8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92f1c000 - 0x92f2afff /usr/lib/libz.1.dylib 0x92fa3000 - 0x9314afff /usr/lib/libSystem.B.dylib 0x98974000 - 0x98977fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6968, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Wed May 4 06:49:05 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x002bb9e0 _solver_triple + 24928 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032d6ef _triple_column + 2479 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x92fa41ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9302b809 _nanosleep + 345 2 libSystem.B.dylib 0x9302b6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x92fd17fd __pthread_start + 345 6 libSystem.B.dylib 0x92fd1682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbda697a8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92f1c000 - 0x92f2afff /usr/lib/libz.1.dylib 0x92fa3000 - 0x9314afff /usr/lib/libSystem.B.dylib 0x98974000 - 0x98977fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=48063, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Wed May 4 10:10:40 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x002bb9e0 _solver_triple + 24928 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032d6ef _triple_column + 2479 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x92fa41ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9302b809 _nanosleep + 345 2 libSystem.B.dylib 0x9302b6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x92fd17fd __pthread_start + 345 6 libSystem.B.dylib 0x92fd1682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbd98b808 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92f1c000 - 0x92f2afff /usr/lib/libz.1.dylib 0x92fa3000 - 0x9314afff /usr/lib/libSystem.B.dylib 0x98974000 - 0x98977fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=48063, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Thu May 5 23:32:35 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x002bb9e0 _solver_triple + 24928 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032d6ef _triple_column + 2479 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x92fa41ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9302b809 _nanosleep + 345 2 libSystem.B.dylib 0x9302b6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x92fd17fd __pthread_start + 345 6 libSystem.B.dylib 0x92fd1682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbd9f2988 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92f1c000 - 0x92f2afff /usr/lib/libz.1.dylib 0x92fa3000 - 0x9314afff /usr/lib/libSystem.B.dylib 0x98974000 - 0x98977fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=74355, iMonCtr=1 Model crash detected, will try to restart... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Fri May 6 06:30:58 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x002bb9e0 _solver_triple + 24928 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032d6ef _triple_column + 2479 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x92fa41ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9302b809 _nanosleep + 345 2 libSystem.B.dylib 0x9302b6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x92fd17fd __pthread_start + 345 6 libSystem.B.dylib 0x92fd1682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbda22298 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92f1c000 - 0x92f2afff /usr/lib/libz.1.dylib 0x92fa3000 - 0x9314afff /usr/lib/libSystem.B.dylib 0x98974000 - 0x98977fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=74355, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(14245,0xa03bc540) malloc: * error for object 0x4038804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14245,0xa03bc540) malloc: * error for object 0x2824c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14245,0xa03bc540) malloc: * error for object 0x4828604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14245,0xa03bc540) malloc: * error for object 0x2824c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(14245,0xa03bc540) malloc: * error for object 0x3828604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14245,0xa03bc540) malloc: * error for object 0x3829804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14245,0xa03bc540) malloc: * error for object 0x3829800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) hadcm3n_6.07_i686-apple-darwin(20987,0xa03bc540) malloc: * error for object 0x3015e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(21639,0xa03bc540) malloc: * error for object 0x3801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21639,0xa03bc540) malloc: * error for object 0x3801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21639,0xa03bc540) malloc: * error for object 0x3801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21639,0xa03bc540) malloc: * error for object 0x3801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21639,0xa03bc540) malloc: * error for object 0x4000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21639,0xa03bc540) malloc: * error for object 0x4000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21639,0xa03bc540) malloc: * error for object 0x4000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21639,0xa03bc540) malloc: * error for object 0x4000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21639,0xa03bc540) malloc: * error for object 0x4000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 May 2011 20:02:13	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	1,036,800	1,290,104	1.2443
09 May 2011 00:35:14	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	1,010,880	1,249,778	1.2363
08 May 2011 07:48:10	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	984,960	1,206,283	1.2247
07 May 2011 17:56:27	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	959,040	1,162,737	1.2124
07 May 2011 00:56:42	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	933,120	1,119,004	1.1992
06 May 2011 09:15:01	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	907,200	1,075,545	1.1856
05 May 2011 15:25:37	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	881,280	1,055,431	1.1976
04 May 2011 22:39:37	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	855,360	1,013,249	1.1846
04 May 2011 09:47:45	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	829,440	969,792	1.1692
03 May 2011 14:30:59	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	803,520	998,381	1.2425
03 May 2011 00:45:45	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	777,600	955,240	1.2284
02 May 2011 08:59:53	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	751,680	916,073	1.2187
01 May 2011 17:07:54	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	725,760	893,446	1.2310
01 May 2011 03:22:00	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	699,840	891,626	1.2740
30 Apr 2011 14:23:20	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	673,920	912,392	1.3539
30 Apr 2011 00:40:58	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	648,000	869,067	1.3412
29 Apr 2011 10:39:18	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	622,080	846,425	1.3606
28 Apr 2011 17:25:16	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	596,160	828,644	1.3900
27 Apr 2011 21:15:23	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	570,240	786,160	1.3786
27 Apr 2011 07:27:56	1136193	12749213	hadcm3n_o6oq_1900_40_007203997_1	544,320	743,744	1.3664