Task 13006359

Name	hadcm3n_o0uk_1940_40_007308697_1
Workunit	7506121
Created	26 Jun 2011, 19:55:53 UTC
Sent	26 Jun 2011, 19:55:58 UTC
Report deadline	26 Sep 2011, 3:23:09 UTC
Received	15 Jul 2011, 15:13:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	942621
Run time	11 days 2 hours 43 min 30 sec
CPU time	7 days 23 hours 0 min 55 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	3.44 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.6.36</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(78850,0xa0aa3540) malloc: * error for object 0x1827804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78850,0xa0aa3540) malloc: * error for object 0x1827800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=78851, selfPID=78851, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x501f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x3024604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x2806604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x2806600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x1814e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x1814e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x1814e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x1814e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x1814e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x1814e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x1814e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x1814e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x1814e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(92714,0xa0aa3540) malloc: * error for object 0x1814e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:19:24 (13852): No heartbeat from core client for 30 sec - exiting 21:19:26 (13852): No heartbeat from core client for 30 sec - exiting 21:19:27 (13852): No heartbeat from core client for 30 sec - exiting 21:19:28 (13852): No heartbeat from core client for 30 sec - exiting 21:19:29 (13852): No heartbeat from core client for 30 sec - exiting 21:19:30 (13852): No heartbeat from core client for 30 sec - exiting 21:19:31 (13852): No heartbeat from core client for 30 sec - exiting 21:19:32 (13852): No heartbeat from core client for 30 sec - exiting 21:19:33 (13852): No heartbeat from core client for 30 sec - exiting 21:19:34 (13852): No heartbeat from core client for 30 sec - exiting 21:19:35 (13852): No heartbeat from core client for 30 sec - exiting 21:19:36 (13852): No heartbeat from core client for 30 sec - exiting 21:19:37 (13852): No heartbeat from core client for 30 sec - exiting 21:19:38 (13852): No heartbeat from core client for 30 sec - exiting 21:19:39 (13852): No heartbeat from core client for 30 sec - exiting 21:19:40 (13852): No heartbeat from core client for 30 sec - exiting 21:19:41 (13852): No heartbeat from core client for 30 sec - exiting 21:19:42 (13852): No heartbeat from core client for 30 sec - exiting 21:19:43 (13852): No heartbeat from core client for 30 sec - exiting 21:19:44 (13852): No heartbeat from core client for 30 sec - exiting 21:19:45 (13852): No heartbeat from core client for 30 sec - exiting 21:19:46 (13852): No heartbeat from core client for 30 sec - exiting 21:19:47 (13852): No heartbeat from core client for 30 sec - exiting 21:19:48 (13852): No heartbeat from core client for 30 sec - exiting 21:19:49 (13852): No heartbeat from core client for 30 sec - exiting 21:19:50 (13852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:28:56 (23584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:32:45 (35760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:32:46 (35760): No heartbeat from core client for 30 sec - exiting Model crashed: TEMPHIST: Write ERROR on history file for namelistNLCFILES tmp/pipe_dummy 2048 05:00:31 (39749): No heartbeat from core client for 30 sec - exiting 05:00:32 (39749): No heartbeat from core client for 30 sec - exiting 05:00:33 (39749): No heartbeat from core client for 30 sec - exiting 05:00:34 (39749): No heartbeat from core client for 30 sec - exiting 05:00:35 (39749): No heartbeat from core client for 30 sec - exiting 05:00:36 (39749): No heartbeat from core client for 30 sec - exiting 05:00:37 (39749): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K540 Mon Jul 11 05:00:40 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x981afc5a ___kill + 10 1 libSystem.B.dylib 0x982425a5 _raise + 26 2 libSystem.B.dylib 0x982586e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9814ec0e _mach_wait_until + 10 1 libSystem.B.dylib 0x981d6429 _nanosleep + 345 2 libSystem.B.dylib 0x981d62ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9817c259 __pthread_start + 345 6 libSystem.B.dylib 0x9817c0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9c8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x944f8000 - 0x94506fff /usr/lib/libz.1.dylib 0x95749000 - 0x9574cfff /usr/lib/system/libmathCommon.A.dylib 0x9814e000 - 0x982f5fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=40292, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K540 Mon Jul 11 05:00:44 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x981afc5a ___kill + 10 1 libSystem.B.dylib 0x982425a5 _raise + 26 2 libSystem.B.dylib 0x982586e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9814ec0e _mach_wait_until + 10 1 libSystem.B.dylib 0x981d6429 _nanosleep + 345 2 libSystem.B.dylib 0x981d62ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9817c259 __pthread_start + 345 6 libSystem.B.dylib 0x9817c0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9c8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x944f8000 - 0x94506fff /usr/lib/libz.1.dylib 0x95749000 - 0x9574cfff /usr/lib/system/libmathCommon.A.dylib 0x9814e000 - 0x982f5fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=40292, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K540 Mon Jul 11 05:00:47 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x981afc5a ___kill + 10 1 libSystem.B.dylib 0x982425a5 _raise + 26 2 libSystem.B.dylib 0x982586e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9814ec0e _mach_wait_until + 10 1 libSystem.B.dylib 0x981d6429 _nanosleep + 345 2 libSystem.B.dylib 0x981d62ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9817c259 __pthread_start + 345 6 libSystem.B.dylib 0x9817c0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9c8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x944f8000 - 0x94506fff /usr/lib/libz.1.dylib 0x95749000 - 0x9574cfff /usr/lib/system/libmathCommon.A.dylib 0x9814e000 - 0x982f5fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=40292, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K540 Mon Jul 11 05:00:51 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x981afc5a ___kill + 10 1 libSystem.B.dylib 0x982425a5 _raise + 26 2 libSystem.B.dylib 0x982586e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9814ec0e _mach_wait_until + 10 1 libSystem.B.dylib 0x981d6429 _nanosleep + 345 2 libSystem.B.dylib 0x981d62ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9817c259 __pthread_start + 345 6 libSystem.B.dylib 0x9817c0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9c8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x944f8000 - 0x94506fff /usr/lib/libz.1.dylib 0x95749000 - 0x9574cfff /usr/lib/system/libmathCommon.A.dylib 0x9814e000 - 0x982f5fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=40292, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K540 Mon Jul 11 05:00:55 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x981afc5a ___kill + 10 1 libSystem.B.dylib 0x982425a5 _raise + 26 2 libSystem.B.dylib 0x982586e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9814ec0e _mach_wait_until + 10 1 libSystem.B.dylib 0x981d6429 _nanosleep + 345 2 libSystem.B.dylib 0x981d62ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9817c259 __pthread_start + 345 6 libSystem.B.dylib 0x9817c0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9c8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x944f8000 - 0x94506fff /usr/lib/libz.1.dylib 0x95749000 - 0x9574cfff /usr/lib/system/libmathCommon.A.dylib 0x9814e000 - 0x982f5fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=40292, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K540 Mon Jul 11 05:00:58 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x981afc5a ___kill + 10 1 libSystem.B.dylib 0x982425a5 _raise + 26 2 libSystem.B.dylib 0x982586e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9814ec0e _mach_wait_until + 10 1 libSystem.B.dylib 0x981d6429 _nanosleep + 345 2 libSystem.B.dylib 0x981d62ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9817c259 __pthread_start + 345 6 libSystem.B.dylib 0x9817c0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9c8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x944f8000 - 0x94506fff /usr/lib/libz.1.dylib 0x95749000 - 0x9574cfff /usr/lib/system/libmathCommon.A.dylib 0x9814e000 - 0x982f5fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=40292, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Jul 2011 03:21:17	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	440,640	686,424	1.5578
10 Jul 2011 02:14:22	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	414,720	649,759	1.5667
08 Jul 2011 22:50:36	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	388,800	606,676	1.5604
07 Jul 2011 22:45:12	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	362,880	565,820	1.5592
07 Jul 2011 15:40:16	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	336,960	523,847	1.5546
07 Jul 2011 15:40:16	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	311,040	482,423	1.5510
05 Jul 2011 08:52:34	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	285,120	442,132	1.5507
04 Jul 2011 16:13:31	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	259,200	402,209	1.5517
03 Jul 2011 22:26:26	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	233,280	363,975	1.5602
03 Jul 2011 04:14:30	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	207,360	324,371	1.5643
02 Jul 2011 04:38:18	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	181,440	279,379	1.5398
01 Jul 2011 12:30:10	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	155,520	239,509	1.5401
30 Jun 2011 20:33:03	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	129,600	198,203	1.5293
30 Jun 2011 00:21:19	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	103,680	157,782	1.5218
29 Jun 2011 01:35:25	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	77,760	117,290	1.5084
28 Jun 2011 05:06:51	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	51,840	77,162	1.4885
27 Jun 2011 11:39:49	942621	13006359	hadcm3n_o0uk_1940_40_007308697_1	25,920	39,998	1.5431