Task 12742550

Name	hadcm3n_o44h_1900_40_007200676_1
Workunit	7398956
Created	28 Mar 2011, 14:09:12 UTC
Sent	30 Mar 2011, 19:07:40 UTC
Report deadline	30 Jun 2011, 2:34:51 UTC
Received	9 May 2011, 23:00:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1136193
Run time	20 days 22 hours 21 min 39 sec
CPU time	15 days 23 hours 44 min 47 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	3.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Fri Apr 1 15:14:49 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x001fd912 _filtr + 4738 1 hadcm3n_um_6.07_i686-apple-darwin 0x00200d33 _ofiltr_cntl + 6291 2 hadcm3n_um_6.07_i686-apple-darwin 0x00305061 _tracer + 42449 3 hadcm3n_um_6.07_i686-apple-darwin 0x0027b525 _rowcalc + 38533 4 hadcm3n_um_6.07_i686-apple-darwin 0x00056cf6 _blokcntl + 12566 5 hadcm3n_um_6.07_i686-apple-darwin 0x0005012b _blokcalc + 16843 6 hadcm3n_um_6.07_i686-apple-darwin 0x00287e62 _row_ctl + 9426 7 hadcm3n_um_6.07_i686-apple-darwin 0x001f9a84 _ocn_ctl + 7252 8 hadcm3n_um_6.07_i686-apple-darwin 0x001f7253 _ocn_for_step + 6755 9 hadcm3n_um_6.07_i686-apple-darwin 0x00380efd _ocn_step + 3693 10 hadcm3n_um_6.07_i686-apple-darwin 0x0039539d _u_model + 45245 11 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 12 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 13 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 14 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbddc0318 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=92290, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Fri Apr 1 18:37:42 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x000c78c2 _dd_call + 1586 1 hadcm3n_um_6.07_i686-apple-darwin 0x000bb0fb _convect + 30699 2 hadcm3n_um_6.07_i686-apple-darwin 0x001115c4 _glue_conv + 724 3 hadcm3n_um_6.07_i686-apple-darwin 0x000bf025 _conv_ctl + 5797 4 hadcm3n_um_6.07_i686-apple-darwin 0x00043f1f _atm_phys + 21855 5 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 6 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 7 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbe0c97a8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1403, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Sun Apr 24 02:26:05 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x0011e4f9 _grey_extinction + 1353 1 hadcm3n_um_6.07_i686-apple-darwin 0x0010e75e _flux_calc + 5102 2 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 3 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 4 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 5 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 6 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 7 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9105f1ae _mach_wait_until + 10 1 libSystem.B.dylib 0x910e6809 _nanosleep + 345 2 libSystem.B.dylib 0x910e66aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9108c7fd __pthread_start + 345 6 libSystem.B.dylib 0x9108c682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbe1c3f68 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9105e000 - 0x91205fff /usr/lib/libSystem.B.dylib 0x92284000 - 0x92287fff /usr/lib/system/libmathCommon.A.dylib 0x94d0b000 - 0x94d19fff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12104, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Mon May 2 18:59:20 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x002bb9e0 _solver_triple + 24928 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032d6ef _triple_column + 2479 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x92fa41ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9302b809 _nanosleep + 345 2 libSystem.B.dylib 0x9302b6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x92fd17fd __pthread_start + 345 6 libSystem.B.dylib 0x92fd1682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbda51ba8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92f1c000 - 0x92f2afff /usr/lib/libz.1.dylib 0x92fa3000 - 0x9314afff /usr/lib/libSystem.B.dylib 0x98974000 - 0x98977fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38026, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Wed May 4 09:57:35 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x002bb9e0 _solver_triple + 24928 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032d6ef _triple_column + 2479 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x92fa41ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9302b809 _nanosleep + 345 2 libSystem.B.dylib 0x9302b6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x92fd17fd __pthread_start + 345 6 libSystem.B.dylib 0x92fd1682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbda22298 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92f1c000 - 0x92f2afff /usr/lib/libz.1.dylib 0x92fa3000 - 0x9314afff /usr/lib/libSystem.B.dylib 0x98974000 - 0x98977fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=48064, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Fri May 6 01:41:08 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x002bb9e0 _solver_triple + 24928 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032d6ef _triple_column + 2479 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x92fa41ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9302b809 _nanosleep + 345 2 libSystem.B.dylib 0x9302b6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x92fd17fd __pthread_start + 345 6 libSystem.B.dylib 0x92fd1682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbda02738 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92f1c000 - 0x92f2afff /usr/lib/libz.1.dylib 0x92fa3000 - 0x9314afff /usr/lib/libSystem.B.dylib 0x98974000 - 0x98977fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=74356, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Sat May 7 03:02:29 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x002bb9e0 _solver_triple + 24928 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032d6ef _triple_column + 2479 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x92fa41ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9302b809 _nanosleep + 345 2 libSystem.B.dylib 0x9302b6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x92fd17fd __pthread_start + 345 6 libSystem.B.dylib 0x92fd1682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbda51ba8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92f1c000 - 0x92f2afff /usr/lib/libz.1.dylib 0x92fa3000 - 0x9314afff /usr/lib/libSystem.B.dylib 0x98974000 - 0x98977fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=86151, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/o44hko.pjd8c10 is not a valid UM file. Error converting file to netcdf: dataout/o44hko.pjd8c10 Error: Input file: dataout/o44hko.pid8c10 is not a valid UM file. Error converting file to netcdf: dataout/o44hko.pid8c10 Error: Input file: dataout/o44hko.pfd8c10 is not a valid UM file. Error converting file to netcdf: dataout/o44hko.pfd8c10 Error: Input file: dataout/o44hka.phd8c10 is not a valid UM file. Error converting file to netcdf: dataout/o44hka.phd8c10 Error: Input file: dataout/o44hka.pgd8c10 is not a valid UM file. Error converting file to netcdf: dataout/o44hka.pgd8c10 Error: Input file: dataout/o44hka.ped8c10 is not a valid UM file. Error converting file to netcdf: dataout/o44hka.ped8c10 Error: Input file: dataout/o44hka.pdd8c10 is not a valid UM file. Error converting file to netcdf: dataout/o44hka.pdd8c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 May 2011 22:24:03	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	1,010,880	1,412,179	1.3970
09 May 2011 05:39:06	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	984,960	1,375,886	1.3969
08 May 2011 15:24:18	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	959,040	1,333,044	1.3900
07 May 2011 21:56:05	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	933,120	1,300,050	1.3932
07 May 2011 07:36:55	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	907,200	1,257,165	1.3858
06 May 2011 16:01:55	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	881,280	1,221,420	1.3860
05 May 2011 20:00:36	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	855,360	1,178,606	1.3779
05 May 2011 03:21:21	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	829,440	1,144,160	1.3794
04 May 2011 14:42:02	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	803,520	1,101,371	1.3707
03 May 2011 19:29:33	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	777,600	1,129,901	1.4531
03 May 2011 06:42:26	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	751,680	1,086,900	1.4460
02 May 2011 18:24:17	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	725,760	1,048,457	1.4446
01 May 2011 22:28:59	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	699,840	1,014,768	1.4500
01 May 2011 09:10:57	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	673,920	971,728	1.4419
30 Apr 2011 20:31:58	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	648,000	928,831	1.4334
30 Apr 2011 06:47:43	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	622,080	885,760	1.4239
29 Apr 2011 18:25:02	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	596,160	842,334	1.4129
28 Apr 2011 22:55:39	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	570,240	799,268	1.4016
28 Apr 2011 03:15:28	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	544,320	756,569	1.3899
27 Apr 2011 14:27:45	1136193	12742550	hadcm3n_o44h_1900_40_007200676_1	518,400	713,938	1.3772