Task 12753305

Name	hadcm3n_o6q3_1900_40_007204046_2
Workunit	7402326
Created	29 Mar 2011, 0:02:20 UTC
Sent	29 Mar 2011, 0:06:59 UTC
Report deadline	28 Jun 2011, 7:34:10 UTC
Received	29 Mar 2011, 20:17:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1136193
Run time	16 hours 43 min 27 sec
CPU time	2 hours 25 min 19 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	3.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> SIGSEGV: segmentation violation Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Tue Mar 29 10:46:39 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x003353a5 _two_coeff_region + 1445 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032cf27 _triple_column + 487 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbdc79508 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=57335, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 SIGILL: illegal instruction Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Tue Mar 29 15:17:17 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x003358a6 _two_coeff_region + 2726 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032cf27 _triple_column + 487 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbdbc7058 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=57335, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGILL: illegal instruction Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Tue Mar 29 21:27:17 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x0011e4ee _grey_extinction + 1342 1 hadcm3n_um_6.07_i686-apple-darwin 0x0010e75e _flux_calc + 5102 2 hadcm3n_um_6.07_i686-apple-darwin 0x001b533e _r2_lwrad + 5054 3 hadcm3n_um_6.07_i686-apple-darwin 0x002513eb _rad_ctl + 39563 4 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 5 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 6 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 7 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbda14df8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=61514, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Wed Mar 30 00:23:57 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x0041be30 _expf.J + 192 1 hadcm3n_um_6.07_i686-apple-darwin 0x0002ce1c _fast_tanh. + 255 2 hadcm3n_um_6.07_i686-apple-darwin 0x0002be15 _ai_calc + 32325 3 hadcm3n_um_6.07_i686-apple-darwin 0x001a2fc4 _isopyc_m + 4068 4 hadcm3n_um_6.07_i686-apple-darwin 0x002fe3b9 _tracer + 14633 5 hadcm3n_um_6.07_i686-apple-darwin 0x0027b525 _rowcalc + 38533 6 hadcm3n_um_6.07_i686-apple-darwin 0x00056cf6 _blokcntl + 12566 7 hadcm3n_um_6.07_i686-apple-darwin 0x0005012b _blokcalc + 16843 8 hadcm3n_um_6.07_i686-apple-darwin 0x00287e62 _row_ctl + 9426 9 hadcm3n_um_6.07_i686-apple-darwin 0x001f9a84 _ocn_ctl + 7252 10 hadcm3n_um_6.07_i686-apple-darwin 0x001f7253 _ocn_for_step + 6755 11 hadcm3n_um_6.07_i686-apple-darwin 0x00380efd _ocn_step + 3693 12 hadcm3n_um_6.07_i686-apple-darwin 0x0039539d _u_model + 45245 13 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 14 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 15 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 16 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbddb7358 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=63548, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Mar 2011 19:19:27	1136193	12753305	hadcm3n_o6q3_1900_40_007204046_2	25,920	24,217	0.9343