Task 12995643

Name	hadcm3n_o646_1940_40_007301789_1
Workunit	7499213
Created	22 Jun 2011, 15:04:58 UTC
Sent	22 Jun 2011, 15:05:08 UTC
Report deadline	21 Sep 2011, 22:32:19 UTC
Received	25 Jul 2011, 2:08:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1151317
Run time	15 days 5 hours 57 min 24 sec
CPU time	14 days 19 hours 28 min 8 sec
Validate state	Invalid
Credit	9,642.24
Device peak FLOPS	3.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:36:54 (29983): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(98707,0xa03bb540) malloc: * error for object 0x3002204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98707,0xa03bb540) malloc: * error for object 0x3002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98707,0xa03bb540) malloc: * error for object 0x3002204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98707,0xa03bb540) malloc: * error for object 0x3002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98707,0xa03bb540) malloc: * error for object 0x3002204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98707,0xa03bb540) malloc: * error for object 0x3002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98707,0xa03bb540) malloc: * error for object 0x3002204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98707,0xa03bb540) malloc: * error for object 0x3002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98707,0xa03bb540) malloc: * error for object 0x3002204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98707,0xa03bb540) malloc: * error for object 0x3002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98707,0xa03bb540) malloc: * error for object 0x4804e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=98708, selfPID=98708, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 131585) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6586, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 131585) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6586, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 131585) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6586, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 131585) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6586, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 131585) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6586, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 131585) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6586, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 22:15:57	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	803,520	1,279,089	1.5919
25 Jul 2011 22:15:57	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	777,600	1,237,905	1.5920
25 Jul 2011 17:37:17	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	751,680	1,197,938	1.5937
25 Jul 2011 16:06:10	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	725,760	1,157,245	1.5945
25 Jul 2011 15:38:49	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	699,840	1,117,840	1.5973
25 Jul 2011 14:54:45	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	673,920	1,078,982	1.6011
25 Jul 2011 13:38:00	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	648,000	1,040,211	1.6053
25 Jul 2011 13:38:00	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	622,080	999,735	1.6071
25 Jul 2011 13:38:00	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	596,160	959,628	1.6097
25 Jul 2011 13:38:00	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	570,240	920,080	1.6135
10 Jul 2011 03:24:48	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	544,320	880,222	1.6171
07 Jul 2011 15:59:21	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	518,400	839,904	1.6202
07 Jul 2011 15:59:21	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	492,480	800,912	1.6263
04 Jul 2011 20:01:49	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	466,560	760,243	1.6295
04 Jul 2011 06:52:06	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	440,640	716,919	1.6270
03 Jul 2011 19:08:44	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	414,720	675,632	1.6291
03 Jul 2011 00:20:46	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	388,800	633,380	1.6291
02 Jul 2011 09:41:37	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	362,880	588,039	1.6205
01 Jul 2011 20:42:47	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	336,960	544,891	1.6171
30 Jun 2011 20:17:50	1151317	12995643	hadcm3n_o646_1940_40_007301789_1	311,040	499,276	1.6052