Task 13293623

Name	hadcm3n_p36j_1940_40_007422165_2
Workunit	7619800
Created	25 Aug 2011, 5:55:00 UTC
Sent	25 Aug 2011, 13:43:51 UTC
Report deadline	24 Nov 2011, 21:11:02 UTC
Received	1 Oct 2011, 23:10:14 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1172754
Run time	19 days 6 hours 28 min 34 sec
CPU time	17 days 19 hours 13 min 11 sec
Validate state	Invalid
Credit	11,197.44
Device peak FLOPS	3.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(53692,0xad2ef2c0) malloc: * error for object 0xc09a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53692,0xad2ef2c0) malloc: * error for object 0xc09a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53692,0xad2ef2c0) malloc: * error for object 0xc09a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53692,0xad2ef2c0) malloc: * error for object 0xc09a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53692,0xad2ef2c0) malloc: * error for object 0xc09a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53692,0xad2ef2c0) malloc: * error for object 0xc09a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53692,0xad2ef2c0) malloc: * error for object 0xc09a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53692,0xad2ef2c0) malloc: * error for object 0xc09a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53692,0xad2ef2c0) malloc: * error for object 0xc09a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53692,0xad2ef2c0) malloc: * error for object 0xc09a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53692,0xad2ef2c0) malloc: * error for object 0x4450a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=53697, selfPID=53697, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(60393,0xad2ef2c0) malloc: * error for object 0x2950004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(60393,0xad2ef2c0) malloc: * error for object 0x2950000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/p36jko.pjf3c10 is not a valid UM file. Error converting file to netcdf: dataout/p36jko.pjf3c10 Error: Input file: dataout/p36jko.pif3c10 is not a valid UM file. Error converting file to netcdf: dataout/p36jko.pif3c10 Error: Input file: dataout/p36jko.pff3c10 is not a valid UM file. Error converting file to netcdf: dataout/p36jko.pff3c10 Error: Input file: dataout/p36jka.phf3c10 is not a valid UM file. Error converting file to netcdf: dataout/p36jka.phf3c10 Error: Input file: dataout/p36jka.pgf3c10 is not a valid UM file. Error converting file to netcdf: dataout/p36jka.pgf3c10 Error: Input file: dataout/p36jka.pef3c10 is not a valid UM file. Error converting file to netcdf: dataout/p36jka.pef3c10 Error: Input file: dataout/p36jka.pdf3c10 is not a valid UM file. Error converting file to netcdf: dataout/p36jka.pdf3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:56:38 (6179): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:56:39 (6179): No heartbeat from core client for 30 sec - exiting 14:56:40 (6179): No heartbeat from core client for 30 sec - exiting 14:56:41 (6179): No heartbeat from core client for 30 sec - exiting 14:56:42 (6179): No heartbeat from core client for 30 sec - exiting 14:56:43 (6179): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Sat Oct 1 14:56:49 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0xcc1000 - 0xceffff /usr/lib/libSystem.B.dylib 0xcfc000 - 0xd0afff /usr/lib/libz.1.dylib 0xd0f000 - 0xd13fff /usr/lib/system/libcache.dylib 0xd18000 - 0xd5bfff /usr/lib/system/libcommonCrypto.dylib 0xd65000 - 0xd68fff /usr/lib/system/libcompiler_rt.dylib 0xd6f000 - 0xd77fff /usr/lib/system/libcopyfile.dylib 0xd7d000 - 0xd8bfff /usr/lib/system/libdispatch.dylib 0xd99000 - 0xd99fff /usr/lib/system/libdnsinfo.dylib 0xd9d000 - 0xd9ffff /usr/lib/system/libdyld.dylib 0xda5000 - 0xda5fff /usr/lib/system/libkeymgr.dylib 0xda9000 - 0xdb1fff /usr/lib/system/liblaunch.dylib 0xdb7000 - 0xdbcfff /usr/lib/system/libmacho.dylib 0xdc1000 - 0xdc4fff /usr/lib/system/libmathCommon.A.dylib 0xdc7000 - 0xdcefff /usr/lib/system/libnotify.dylib 0xdd5000 - 0xdd6fff /usr/lib/system/libquarantine.dylib 0xddb000 - 0xddcfff /usr/lib/system/libremovefile.dylib 0xde1000 - 0xde2fff /usr/lib/system/libsystem_blocks.dylib 0xde7000 - 0xeb2fff /usr/lib/system/libsystem_c.dylib 0xeec000 - 0xef3fff /usr/lib/system/libsystem_dnssd.dylib 0xef9000 - 0xf29fff /usr/lib/system/libsystem_info.dylib 0xf3d000 - 0xf5bfff /usr/lib/system/libsystem_kernel.dylib 0xf6b000 - 0xf6ffff /usr/lib/system/libsystem_network.dylib 0xf76000 - 0xf77fff /usr/lib/system/libsystem_sandbox.dylib 0xf7c000 - 0xf7dfff /usr/lib/system/libunc.dylib 0xf82000 - 0xf8afff /usr/lib/system/libunwind.dylib 0xf91000 - 0xfa7fff /usr/lib/system/libxpc.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6553, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Sat Oct 1 14:56:51 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0xcc1000 - 0xceffff /usr/lib/libSystem.B.dylib 0xcfc000 - 0xd0afff /usr/lib/libz.1.dylib 0xd0f000 - 0xd13fff /usr/lib/system/libcache.dylib 0xd18000 - 0xd5bfff /usr/lib/system/libcommonCrypto.dylib 0xd65000 - 0xd68fff /usr/lib/system/libcompiler_rt.dylib 0xd6f000 - 0xd77fff /usr/lib/system/libcopyfile.dylib 0xd7d000 - 0xd8bfff /usr/lib/system/libdispatch.dylib 0xd99000 - 0xd99fff /usr/lib/system/libdnsinfo.dylib 0xd9d000 - 0xd9ffff /usr/lib/system/libdyld.dylib 0xda5000 - 0xda5fff /usr/lib/system/libkeymgr.dylib 0xda9000 - 0xdb1fff /usr/lib/system/liblaunch.dylib 0xdb7000 - 0xdbcfff /usr/lib/system/libmacho.dylib 0xdc1000 - 0xdc4fff /usr/lib/system/libmathCommon.A.dylib 0xdc7000 - 0xdcefff /usr/lib/system/libnotify.dylib 0xdd5000 - 0xdd6fff /usr/lib/system/libquarantine.dylib 0xddb000 - 0xddcfff /usr/lib/system/libremovefile.dylib 0xde1000 - 0xde2fff /usr/lib/system/libsystem_blocks.dylib 0xde7000 - 0xeb2fff /usr/lib/system/libsystem_c.dylib 0xeec000 - 0xef3fff /usr/lib/system/libsystem_dnssd.dylib 0xef9000 - 0xf29fff /usr/lib/system/libsystem_info.dylib 0xf3d000 - 0xf5bfff /usr/lib/system/libsystem_kernel.dylib 0xf6b000 - 0xf6ffff /usr/lib/system/libsystem_network.dylib 0xf76000 - 0xf77fff /usr/lib/system/libsystem_sandbox.dylib 0xf7c000 - 0xf7dfff /usr/lib/system/libunc.dylib 0xf82000 - 0xf8afff /usr/lib/system/libunwind.dylib 0xf91000 - 0xfa7fff /usr/lib/system/libxpc.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6553, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Sat Oct 1 14:56:54 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0xcc1000 - 0xceffff /usr/lib/libSystem.B.dylib 0xcfc000 - 0xd0afff /usr/lib/libz.1.dylib 0xd0f000 - 0xd13fff /usr/lib/system/libcache.dylib 0xd18000 - 0xd5bfff /usr/lib/system/libcommonCrypto.dylib 0xd65000 - 0xd68fff /usr/lib/system/libcompiler_rt.dylib 0xd6f000 - 0xd77fff /usr/lib/system/libcopyfile.dylib 0xd7d000 - 0xd8bfff /usr/lib/system/libdispatch.dylib 0xd99000 - 0xd99fff /usr/lib/system/libdnsinfo.dylib 0xd9d000 - 0xd9ffff /usr/lib/system/libdyld.dylib 0xda5000 - 0xda5fff /usr/lib/system/libkeymgr.dylib 0xda9000 - 0xdb1fff /usr/lib/system/liblaunch.dylib 0xdb7000 - 0xdbcfff /usr/lib/system/libmacho.dylib 0xdc1000 - 0xdc4fff /usr/lib/system/libmathCommon.A.dylib 0xdc7000 - 0xdcefff /usr/lib/system/libnotify.dylib 0xdd5000 - 0xdd6fff /usr/lib/system/libquarantine.dylib 0xddb000 - 0xddcfff /usr/lib/system/libremovefile.dylib 0xde1000 - 0xde2fff /usr/lib/system/libsystem_blocks.dylib 0xde7000 - 0xeb2fff /usr/lib/system/libsystem_c.dylib 0xeec000 - 0xef3fff /usr/lib/system/libsystem_dnssd.dylib 0xef9000 - 0xf29fff /usr/lib/system/libsystem_info.dylib 0xf3d000 - 0xf5bfff /usr/lib/system/libsystem_kernel.dylib 0xf6b000 - 0xf6ffff /usr/lib/system/libsystem_network.dylib 0xf76000 - 0xf77fff /usr/lib/system/libsystem_sandbox.dylib 0xf7c000 - 0xf7dfff /usr/lib/system/libunc.dylib 0xf82000 - 0xf8afff /usr/lib/system/libunwind.dylib 0xf91000 - 0xfa7fff /usr/lib/system/libxpc.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6553, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Sat Oct 1 14:56:56 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0xcc1000 - 0xceffff /usr/lib/libSystem.B.dylib 0xcfc000 - 0xd0afff /usr/lib/libz.1.dylib 0xd0f000 - 0xd13fff /usr/lib/system/libcache.dylib 0xd18000 - 0xd5bfff /usr/lib/system/libcommonCrypto.dylib 0xd65000 - 0xd68fff /usr/lib/system/libcompiler_rt.dylib 0xd6f000 - 0xd77fff /usr/lib/system/libcopyfile.dylib 0xd7d000 - 0xd8bfff /usr/lib/system/libdispatch.dylib 0xd99000 - 0xd99fff /usr/lib/system/libdnsinfo.dylib 0xd9d000 - 0xd9ffff /usr/lib/system/libdyld.dylib 0xda5000 - 0xda5fff /usr/lib/system/libkeymgr.dylib 0xda9000 - 0xdb1fff /usr/lib/system/liblaunch.dylib 0xdb7000 - 0xdbcfff /usr/lib/system/libmacho.dylib 0xdc1000 - 0xdc4fff /usr/lib/system/libmathCommon.A.dylib 0xdc7000 - 0xdcefff /usr/lib/system/libnotify.dylib 0xdd5000 - 0xdd6fff /usr/lib/system/libquarantine.dylib 0xddb000 - 0xddcfff /usr/lib/system/libremovefile.dylib 0xde1000 - 0xde2fff /usr/lib/system/libsystem_blocks.dylib 0xde7000 - 0xeb2fff /usr/lib/system/libsystem_c.dylib 0xeec000 - 0xef3fff /usr/lib/system/libsystem_dnssd.dylib 0xef9000 - 0xf29fff /usr/lib/system/libsystem_info.dylib 0xf3d000 - 0xf5bfff /usr/lib/system/libsystem_kernel.dylib 0xf6b000 - 0xf6ffff /usr/lib/system/libsystem_network.dylib 0xf76000 - 0xf77fff /usr/lib/system/libsystem_sandbox.dylib 0xf7c000 - 0xf7dfff /usr/lib/system/libunc.dylib 0xf82000 - 0xf8afff /usr/lib/system/libunwind.dylib 0xf91000 - 0xfa7fff /usr/lib/system/libxpc.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6553, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Sat Oct 1 14:56:59 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0xcc1000 - 0xceffff /usr/lib/libSystem.B.dylib 0xcfc000 - 0xd0afff /usr/lib/libz.1.dylib 0xd0f000 - 0xd13fff /usr/lib/system/libcache.dylib 0xd18000 - 0xd5bfff /usr/lib/system/libcommonCrypto.dylib 0xd65000 - 0xd68fff /usr/lib/system/libcompiler_rt.dylib 0xd6f000 - 0xd77fff /usr/lib/system/libcopyfile.dylib 0xd7d000 - 0xd8bfff /usr/lib/system/libdispatch.dylib 0xd99000 - 0xd99fff /usr/lib/system/libdnsinfo.dylib 0xd9d000 - 0xd9ffff /usr/lib/system/libdyld.dylib 0xda5000 - 0xda5fff /usr/lib/system/libkeymgr.dylib 0xda9000 - 0xdb1fff /usr/lib/system/liblaunch.dylib 0xdb7000 - 0xdbcfff /usr/lib/system/libmacho.dylib 0xdc1000 - 0xdc4fff /usr/lib/system/libmathCommon.A.dylib 0xdc7000 - 0xdcefff /usr/lib/system/libnotify.dylib 0xdd5000 - 0xdd6fff /usr/lib/system/libquarantine.dylib 0xddb000 - 0xddcfff /usr/lib/system/libremovefile.dylib 0xde1000 - 0xde2fff /usr/lib/system/libsystem_blocks.dylib 0xde7000 - 0xeb2fff /usr/lib/system/libsystem_c.dylib 0xeec000 - 0xef3fff /usr/lib/system/libsystem_dnssd.dylib 0xef9000 - 0xf29fff /usr/lib/system/libsystem_info.dylib 0xf3d000 - 0xf5bfff /usr/lib/system/libsystem_kernel.dylib 0xf6b000 - 0xf6ffff /usr/lib/system/libsystem_network.dylib 0xf76000 - 0xf77fff /usr/lib/system/libsystem_sandbox.dylib 0xf7c000 - 0xf7dfff /usr/lib/system/libunc.dylib 0xf82000 - 0xf8afff /usr/lib/system/libunwind.dylib 0xf91000 - 0xfa7fff /usr/lib/system/libxpc.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6553, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Sat Oct 1 14:57:02 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0xcc1000 - 0xceffff /usr/lib/libSystem.B.dylib 0xcfc000 - 0xd0afff /usr/lib/libz.1.dylib 0xd0f000 - 0xd13fff /usr/lib/system/libcache.dylib 0xd18000 - 0xd5bfff /usr/lib/system/libcommonCrypto.dylib 0xd65000 - 0xd68fff /usr/lib/system/libcompiler_rt.dylib 0xd6f000 - 0xd77fff /usr/lib/system/libcopyfile.dylib 0xd7d000 - 0xd8bfff /usr/lib/system/libdispatch.dylib 0xd99000 - 0xd99fff /usr/lib/system/libdnsinfo.dylib 0xd9d000 - 0xd9ffff /usr/lib/system/libdyld.dylib 0xda5000 - 0xda5fff /usr/lib/system/libkeymgr.dylib 0xda9000 - 0xdb1fff /usr/lib/system/liblaunch.dylib 0xdb7000 - 0xdbcfff /usr/lib/system/libmacho.dylib 0xdc1000 - 0xdc4fff /usr/lib/system/libmathCommon.A.dylib 0xdc7000 - 0xdcefff /usr/lib/system/libnotify.dylib 0xdd5000 - 0xdd6fff /usr/lib/system/libquarantine.dylib 0xddb000 - 0xddcfff /usr/lib/system/libremovefile.dylib 0xde1000 - 0xde2fff /usr/lib/system/libsystem_blocks.dylib 0xde7000 - 0xeb2fff /usr/lib/system/libsystem_c.dylib 0xeec000 - 0xef3fff /usr/lib/system/libsystem_dnssd.dylib 0xef9000 - 0xf29fff /usr/lib/system/libsystem_info.dylib 0xf3d000 - 0xf5bfff /usr/lib/system/libsystem_kernel.dylib 0xf6b000 - 0xf6ffff /usr/lib/system/libsystem_network.dylib 0xf76000 - 0xf77fff /usr/lib/system/libsystem_sandbox.dylib 0xf7c000 - 0xf7dfff /usr/lib/system/libunc.dylib 0xf82000 - 0xf8afff /usr/lib/system/libunwind.dylib 0xf91000 - 0xfa7fff /usr/lib/system/libxpc.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6553, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Sep 2011 04:32:12	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	933,120	1,532,450	1.6423
29 Sep 2011 23:07:05	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	907,200	1,491,871	1.6445
29 Sep 2011 01:47:17	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	881,280	1,450,828	1.6463
28 Sep 2011 23:09:57	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	855,360	1,405,964	1.6437
27 Sep 2011 23:07:17	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	829,440	1,368,355	1.6497
26 Sep 2011 23:06:08	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	803,520	1,326,691	1.6511
26 Sep 2011 06:18:13	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	777,600	1,287,262	1.6554
25 Sep 2011 23:08:25	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	751,680	1,245,274	1.6567
24 Sep 2011 23:08:04	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	725,760	1,201,144	1.6550
24 Sep 2011 03:48:47	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	699,840	1,163,476	1.6625
23 Sep 2011 23:10:08	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	673,920	1,127,010	1.6723
23 Sep 2011 03:01:28	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	648,000	1,089,860	1.6819
22 Sep 2011 23:07:51	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	622,080	1,047,259	1.6835
21 Sep 2011 23:06:30	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	596,160	1,002,158	1.6810
21 Sep 2011 05:14:19	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	570,240	964,102	1.6907
20 Sep 2011 23:10:10	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	544,320	923,178	1.6960
19 Sep 2011 23:05:47	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	518,400	885,984	1.7091
18 Sep 2011 23:05:58	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	492,480	840,440	1.7065
18 Sep 2011 05:17:57	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	466,560	797,372	1.7090
17 Sep 2011 23:08:57	1165424	13293623	hadcm3n_p36j_1940_40_007422165_2	440,640	755,577	1.7147