Task 15134319

Name	hadcm3n_o147_2100_40_008154348_0
Workunit	8309472
Created	17 Aug 2012, 11:35:47 UTC
Sent	17 Aug 2012, 11:40:40 UTC
Report deadline	16 Nov 2012, 19:07:51 UTC
Received	4 Oct 2012, 7:30:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1179081
Run time	1 days 17 hours 47 min 28 sec
CPU time	9 hours 47 min 39 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	3.22 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> SIGILL: illegal instruction Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Sat Aug 18 01:59:30 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbde1fc88 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=57764, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Sat Aug 18 04:44:19 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbde1fc88 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=57764, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Sun Aug 19 01:58:58 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbdece648 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69503, iMonCtr=1 Model crash detected, will try to restart... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Sun Aug 19 02:15:08 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbdd918d8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69503, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Sun Aug 19 04:34:37 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbdd0ce08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69503, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Sun Aug 19 06:42:38 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbdd0ce08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69503, iMonCtr=1 Model crash detected, will try to restart... SIGILL: illegal instruction Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Sun Aug 19 12:54:01 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbde1fc88 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69503, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Sun Aug 19 22:21:20 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbdd918d8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=78827, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 SIGSEGV: segmentation violation Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Mon Aug 20 17:18:32 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbe0e37e8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=82615, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Tue Aug 21 00:10:57 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbde1fc88 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=82615, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Tue Aug 21 11:33:33 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbdd918d8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=87488, iMonCtr=1 Model crash detected, will try to restart... SIGILL: illegal instruction Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Tue Aug 21 11:56:54 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbde1fc88 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90396000 - 0x90397fff /usr/lib/system/libunc.dylib 0x909c3000 - 0x909c7fff /usr/lib/system/libcache.dylib 0x90f17000 - 0x90f1afff /usr/lib/system/libmathCommon.A.dylib 0x913be000 - 0x913ecfff /usr/lib/libSystem.B.dylib 0x913fa000 - 0x913fdfff /usr/lib/system/libcompiler_rt.dylib 0x91440000 - 0x91448fff /usr/lib/system/libcopyfile.dylib 0x91ad0000 - 0x91ad4fff /usr/lib/system/libsystem_network.dylib 0x91cb8000 - 0x91cbafff /usr/lib/system/libdyld.dylib 0x92709000 - 0x9270afff /usr/lib/system/libquarantine.dylib 0x937fb000 - 0x93811fff /usr/lib/system/libxpc.dylib 0x93a3a000 - 0x93a48fff /usr/lib/libz.1.dylib 0x946ce000 - 0x946cffff /usr/lib/system/libremovefile.dylib 0x946d0000 - 0x946d7fff /usr/lib/system/libsystem_dnssd.dylib 0x946e0000 - 0x946fefff /usr/lib/system/libsystem_kernel.dylib 0x95a0c000 - 0x95a4ffff /usr/lib/system/libcommonCrypto.dylib 0x96ad8000 - 0x96ad8fff /usr/lib/system/libkeymgr.dylib 0x97570000 - 0x9759ffff /usr/lib/system/libsystem_info.dylib 0x97c3a000 - 0x97c3afff /usr/lib/system/libdnsinfo.dylib 0x9867c000 - 0x98684fff /usr/lib/system/libunwind.dylib 0x98d36000 - 0x98d3dfff /usr/lib/system/libsystem_notify.dylib 0x98d3f000 - 0x98d47fff /usr/lib/system/liblaunch.dylib 0x98d4a000 - 0x98d4ffff /usr/lib/system/libmacho.dylib 0x9ac1e000 - 0x9ac2cfff /usr/lib/system/libdispatch.dylib 0x9b593000 - 0x9b65efff /usr/lib/system/libsystem_c.dylib 0x9b998000 - 0x9b999fff /usr/lib/system/libsystem_sandbox.dylib 0x9cb67000 - 0x9cb68fff /usr/lib/system/libsystem_blocks.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=87488, iMonCtr=1 Model crash detected, will try to restart... Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: TRANSO2A: Missing data in ocean UV fields tmp/pipe_dummy 2048 Model crashed: TRANSO2A: Missing data in ocean UV fields tmp/pipe_dummy 2048 Model crashed: TRANSO2A: Missing data in ocean UV fields tmp/pipe_dummy 2048 Model crashed: TRANSO2A: Missing data in ocean UV fields tmp/pipe_dummy 2048 Model crashed: TRANSO2A: Missing data in ocean UV fields tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Aug 2012 09:34:37	1179081	15134319	hadcm3n_o147_2100_40_008154348_0	77,760	40,489	0.5207
19 Aug 2012 01:32:48	1179081	15134319	hadcm3n_o147_2100_40_008154348_0	51,840	33,420	0.6447
17 Aug 2012 23:07:16	1179081	15134319	hadcm3n_o147_2100_40_008154348_0	25,920	11,407	0.4401