Task 13959062

Name	hadcm3n_yerq_1980_40_007694717_0
Workunit	7849825
Created	24 Jan 2012, 1:23:26 UTC
Sent	24 Jan 2012, 1:32:05 UTC
Report deadline	24 Apr 2012, 8:59:16 UTC
Received	18 Feb 2012, 11:02:05 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1182653
Run time	8 days 10 hours 17 min 33 sec
CPU time	5 days 22 hours 37 min 45 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.94 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SIGILL: illegal instruction Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Sat Feb 18 11:59:49 2012 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x96f488c2 ecx: 0xbea8738c edx: 0x96f14266 edi: 0x00000000 esi: 0x00000000 ebp: 0xbea873c8 esp: 0xbea8738c ss: 0x0000001f efl: 0x00000206 eip: 0x96f14266 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92022000 - 0x92026fff /usr/lib/system/libmathCommon.A.dylib 0x924dd000 - 0x924ebfff /usr/lib/libz.1.dylib 0x96b44000 - 0x96b4bfff /usr/lib/libgcc_s.1.dylib 0x96f13000 - 0x9707afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=135, iMonCtr=1 Model crash detected, will try to restart... SIGILL: illegal instruction Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Sat Feb 18 11:59:53 2012 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x96f488c2 ecx: 0xbea8738c edx: 0x96f14266 edi: 0x00000000 esi: 0x00000000 ebp: 0xbea873c8 esp: 0xbea8738c ss: 0x0000001f efl: 0x00000206 eip: 0x96f14266 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92022000 - 0x92026fff /usr/lib/system/libmathCommon.A.dylib 0x924dd000 - 0x924ebfff /usr/lib/libz.1.dylib 0x96b44000 - 0x96b4bfff /usr/lib/libgcc_s.1.dylib 0x96f13000 - 0x9707afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=135, iMonCtr=1 Model crash detected, will try to restart... SIGILL: illegal instruction Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Sat Feb 18 11:59:57 2012 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x96f488c2 ecx: 0xbea8738c edx: 0x96f14266 edi: 0x00000000 esi: 0x00000000 ebp: 0xbea873c8 esp: 0xbea8738c ss: 0x0000001f efl: 0x00000206 eip: 0x96f14266 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92022000 - 0x92026fff /usr/lib/system/libmathCommon.A.dylib 0x924dd000 - 0x924ebfff /usr/lib/libz.1.dylib 0x96b44000 - 0x96b4bfff /usr/lib/libgcc_s.1.dylib 0x96f13000 - 0x9707afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=135, iMonCtr=1 Model crash detected, will try to restart... SIGILL: illegal instruction Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Sat Feb 18 12:00:01 2012 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x96f488c2 ecx: 0xbea8738c edx: 0x96f14266 edi: 0x00000000 esi: 0x00000000 ebp: 0xbea873c8 esp: 0xbea8738c ss: 0x0000001f efl: 0x00000206 eip: 0x96f14266 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92022000 - 0x92026fff /usr/lib/system/libmathCommon.A.dylib 0x924dd000 - 0x924ebfff /usr/lib/libz.1.dylib 0x96b44000 - 0x96b4bfff /usr/lib/libgcc_s.1.dylib 0x96f13000 - 0x9707afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=135, iMonCtr=1 Model crash detected, will try to restart... SIGILL: illegal instruction Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Sat Feb 18 12:00:05 2012 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x96f488c2 ecx: 0xbea8738c edx: 0x96f14266 edi: 0x00000000 esi: 0x00000000 ebp: 0xbea873c8 esp: 0xbea8738c ss: 0x0000001f efl: 0x00000206 eip: 0x96f14266 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92022000 - 0x92026fff /usr/lib/system/libmathCommon.A.dylib 0x924dd000 - 0x924ebfff /usr/lib/libz.1.dylib 0x96b44000 - 0x96b4bfff /usr/lib/libgcc_s.1.dylib 0x96f13000 - 0x9707afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=135, iMonCtr=1 Model crash detected, will try to restart... SIGILL: illegal instruction Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Sat Feb 18 12:00:09 2012 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x96f488c2 ecx: 0xbea8738c edx: 0x96f14266 edi: 0x00000000 esi: 0x00000000 ebp: 0xbea873c8 esp: 0xbea8738c ss: 0x0000001f efl: 0x00000206 eip: 0x96f14266 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92022000 - 0x92026fff /usr/lib/system/libmathCommon.A.dylib 0x924dd000 - 0x924ebfff /usr/lib/libz.1.dylib 0x96b44000 - 0x96b4bfff /usr/lib/libgcc_s.1.dylib 0x96f13000 - 0x9707afff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=135, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Feb 2012 21:54:15	1182653	13959062	hadcm3n_yerq_1980_40_007694717_0	259,200	638,218	2.4623
17 Feb 2012 03:09:27	1182653	13959062	hadcm3n_yerq_1980_40_007694717_0	233,280	574,555	2.4629
15 Feb 2012 14:54:11	1182653	13959062	hadcm3n_yerq_1980_40_007694717_0	207,360	510,340	2.4611
02 Feb 2012 12:25:47	1182653	13959062	hadcm3n_yerq_1980_40_007694717_0	181,440	446,292	2.4597
01 Feb 2012 05:05:55	1182653	13959062	hadcm3n_yerq_1980_40_007694717_0	155,520	382,538	2.4597
30 Jan 2012 18:13:41	1182653	13959062	hadcm3n_yerq_1980_40_007694717_0	129,600	318,635	2.4586
29 Jan 2012 23:23:26	1182653	13959062	hadcm3n_yerq_1980_40_007694717_0	103,680	255,320	2.4626
28 Jan 2012 20:34:03	1182653	13959062	hadcm3n_yerq_1980_40_007694717_0	77,760	191,315	2.4603
27 Jan 2012 18:10:22	1182653	13959062	hadcm3n_yerq_1980_40_007694717_0	51,840	126,828	2.4465
26 Jan 2012 00:03:17	1182653	13959062	hadcm3n_yerq_1980_40_007694717_0	25,920	63,102	2.4345