Task 13328490

Name	hadcm3n_o45z_1940_40_007435682_2
Workunit	7633185
Created	2 Sep 2011, 7:30:23 UTC
Sent	2 Sep 2011, 7:36:30 UTC
Report deadline	2 Dec 2011, 15:03:41 UTC
Received	6 Sep 2011, 6:08:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	992791
Run time	18 hours 21 min 21 sec
CPU time	17 hours 6 min 33 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	3.44 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.6.36</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 15:24:48 (2341): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:29:19 (5840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:33:49 (5868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:38:21 (5896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:42:52 (5924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:47:23 (5952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:51:54 (5978): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:56:25 (6006): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:00:56 (6030): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:05:26 (6059): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:09:58 (6085): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:14:29 (6113): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:19:00 (6137): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:23:32 (6165): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:28:02 (6191): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:32:33 (6221): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:37:04 (6245): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:41:35 (6277): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:46:06 (6303): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:37 (6331): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:08 (6355): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:59:39 (6383): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:04:10 (6409): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:08:41 (6438): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:13:12 (6462): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:17:43 (6490): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:22:14 (6516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:26:46 (6544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:16 (6570): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:35:48 (6598): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:40:18 (6628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:44:50 (6656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:20 (6680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:53:52 (6708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:22 (6734): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:54 (6762): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:25 (6787): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:56 (6817): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:16:26 (6843): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Tue Sep 6 08:05:07 2011 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. 0 hadcm3n_um_6.07_i686-apple-darwin 0x00445d97 __intel_new_memset + 63047 1 hadcm3n_um_6.07_i686-apple-darwin 0x0043bd7a __intel_new_memset + 22058 2 libSystem.B.dylib 0x955601cb _sigtramp + 43 3 ??? 0xffffffff 0x0 + 4294967295 4 libSystem.B.dylib 0x955d4242 raise + 26 5 libSystem.B.dylib 0x955e0681 abort + 73 6 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 7 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb start + 251 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 11 ??? 0x00000003 0x0 + 3 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x955287c2 ecx: 0xbfffd4ec edx: 0x954f4166 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd528 esp: 0xbfffd4ec ss: 0x0000001f efl: 0x00000206 eip: 0x954f4166 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x919c0000 - 0x919c4fff /usr/lib/system/libmathCommon.A.dylib 0x9353f000 - 0x9354dfff /usr/lib/libz.1.dylib 0x9493b000 - 0x94942fff /usr/lib/libgcc_s.1.dylib 0x954f3000 - 0x9565afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=138, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Tue Sep 6 08:05:10 2011 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. 0 hadcm3n_um_6.07_i686-apple-darwin 0x00445d97 __intel_new_memset + 63047 1 hadcm3n_um_6.07_i686-apple-darwin 0x0043bd7a __intel_new_memset + 22058 2 libSystem.B.dylib 0x955601cb _sigtramp + 43 3 ??? 0xffffffff 0x0 + 4294967295 4 libSystem.B.dylib 0x955d4242 raise + 26 5 libSystem.B.dylib 0x955e0681 abort + 73 6 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 7 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb start + 251 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 11 ??? 0x00000003 0x0 + 3 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x955287c2 ecx: 0xbfffd4ec edx: 0x954f4166 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd528 esp: 0xbfffd4ec ss: 0x0000001f efl: 0x00000206 eip: 0x954f4166 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x919c0000 - 0x919c4fff /usr/lib/system/libmathCommon.A.dylib 0x9353f000 - 0x9354dfff /usr/lib/libz.1.dylib 0x9493b000 - 0x94942fff /usr/lib/libgcc_s.1.dylib 0x954f3000 - 0x9565afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=138, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Tue Sep 6 08:05:12 2011 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. 0 hadcm3n_um_6.07_i686-apple-darwin 0x00445d97 __intel_new_memset + 63047 1 hadcm3n_um_6.07_i686-apple-darwin 0x0043bd7a __intel_new_memset + 22058 2 libSystem.B.dylib 0x955601cb _sigtramp + 43 3 ??? 0xffffffff 0x0 + 4294967295 4 libSystem.B.dylib 0x955d4242 raise + 26 5 libSystem.B.dylib 0x955e0681 abort + 73 6 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 7 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb start + 251 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 11 ??? 0x00000003 0x0 + 3 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x955287c2 ecx: 0xbfffd4ec edx: 0x954f4166 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd528 esp: 0xbfffd4ec ss: 0x0000001f efl: 0x00000206 eip: 0x954f4166 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x919c0000 - 0x919c4fff /usr/lib/system/libmathCommon.A.dylib 0x9353f000 - 0x9354dfff /usr/lib/libz.1.dylib 0x9493b000 - 0x94942fff /usr/lib/libgcc_s.1.dylib 0x954f3000 - 0x9565afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=138, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Tue Sep 6 08:05:15 2011 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. 0 hadcm3n_um_6.07_i686-apple-darwin 0x00445d97 __intel_new_memset + 63047 1 hadcm3n_um_6.07_i686-apple-darwin 0x0043bd7a __intel_new_memset + 22058 2 libSystem.B.dylib 0x955601cb _sigtramp + 43 3 ??? 0xffffffff 0x0 + 4294967295 4 libSystem.B.dylib 0x955d4242 raise + 26 5 libSystem.B.dylib 0x955e0681 abort + 73 6 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 7 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb start + 251 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 11 ??? 0x00000003 0x0 + 3 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x955287c2 ecx: 0xbfffd4ec edx: 0x954f4166 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd528 esp: 0xbfffd4ec ss: 0x0000001f efl: 0x00000206 eip: 0x954f4166 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x919c0000 - 0x919c4fff /usr/lib/system/libmathCommon.A.dylib 0x9353f000 - 0x9354dfff /usr/lib/libz.1.dylib 0x9493b000 - 0x94942fff /usr/lib/libgcc_s.1.dylib 0x954f3000 - 0x9565afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=138, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Tue Sep 6 08:05:18 2011 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. 0 hadcm3n_um_6.07_i686-apple-darwin 0x00445d97 __intel_new_memset + 63047 1 hadcm3n_um_6.07_i686-apple-darwin 0x0043bd7a __intel_new_memset + 22058 2 libSystem.B.dylib 0x955601cb _sigtramp + 43 3 ??? 0xffffffff 0x0 + 4294967295 4 libSystem.B.dylib 0x955d4242 raise + 26 5 libSystem.B.dylib 0x955e0681 abort + 73 6 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 7 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb start + 251 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 11 ??? 0x00000003 0x0 + 3 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x955287c2 ecx: 0xbfffd4ec edx: 0x954f4166 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd528 esp: 0xbfffd4ec ss: 0x0000001f efl: 0x00000206 eip: 0x954f4166 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x919c0000 - 0x919c4fff /usr/lib/system/libmathCommon.A.dylib 0x9353f000 - 0x9354dfff /usr/lib/libz.1.dylib 0x9493b000 - 0x94942fff /usr/lib/libgcc_s.1.dylib 0x954f3000 - 0x9565afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=138, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Tue Sep 6 08:05:20 2011 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. 0 hadcm3n_um_6.07_i686-apple-darwin 0x00445d97 __intel_new_memset + 63047 1 hadcm3n_um_6.07_i686-apple-darwin 0x0043bd7a __intel_new_memset + 22058 2 libSystem.B.dylib 0x955601cb _sigtramp + 43 3 ??? 0xffffffff 0x0 + 4294967295 4 libSystem.B.dylib 0x955d4242 raise + 26 5 libSystem.B.dylib 0x955e0681 abort + 73 6 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 7 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb start + 251 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 11 ??? 0x00000003 0x0 + 3 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x955287c2 ecx: 0xbfffd4ec edx: 0x954f4166 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd528 esp: 0xbfffd4ec ss: 0x0000001f efl: 0x00000206 eip: 0x954f4166 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x919c0000 - 0x919c4fff /usr/lib/system/libmathCommon.A.dylib 0x9353f000 - 0x9354dfff /usr/lib/libz.1.dylib 0x9493b000 - 0x94942fff /usr/lib/libgcc_s.1.dylib 0x954f3000 - 0x9565afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=138, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Sep 2011 06:53:13	992791	13328490	hadcm3n_o45z_1940_40_007435682_2	25,920	38,058	1.4683