Task 12926333

Name	hadcm3n_o66p_1940_40_007266495_0
Workunit	7464735
Created	2 Jun 2011, 21:27:18 UTC
Sent	2 Jun 2011, 21:27:24 UTC
Report deadline	2 Sep 2011, 4:54:35 UTC
Received	22 Jul 2011, 15:05:15 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1018716
Run time	12 days 10 hours 54 min 59 sec
CPU time	10 days 13 hours 56 min 23 sec
Validate state	Invalid
Credit	9,642.24
Device peak FLOPS	3.76 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:10:48 (352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:07:11 (413): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:18:01 (401): No heartbeat from core client for 30 sec - exiting 20:18:02 (401): No heartbeat from core client for 30 sec - exiting 20:18:03 (401): No heartbeat from core client for 30 sec - exiting 20:18:04 (401): No heartbeat from core client for 30 sec - exiting 20:18:05 (401): No heartbeat from core client for 30 sec - exiting 20:18:06 (401): No heartbeat from core client for 30 sec - exiting 20:18:07 (401): No heartbeat from core client for 30 sec - exiting 20:18:08 (401): No heartbeat from core client for 30 sec - exiting 20:18:09 (401): No heartbeat from core client for 30 sec - exiting 20:18:10 (401): No heartbeat from core client for 30 sec - exiting 20:18:11 (401): No heartbeat from core client for 30 sec - exiting 20:18:13 (401): No heartbeat from core client for 30 sec - exiting 20:18:14 (401): No heartbeat from core client for 30 sec - exiting 20:18:15 (401): No heartbeat from core client for 30 sec - exiting 20:18:16 (401): No heartbeat from core client for 30 sec - exiting 20:18:17 (401): No heartbeat from core client for 30 sec - exiting 20:18:18 (401): No heartbeat from core client for 30 sec - exiting 20:18:19 (401): No heartbeat from core client for 30 sec - exiting 20:18:20 (401): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:58:05 (995): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:58:27 (995): No heartbeat from core client for 30 sec - exiting 11:58:28 (995): No heartbeat from core client for 30 sec - exiting 11:58:29 (995): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1047004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x1062600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7718,0xa097b540) malloc: * error for object 0x84ae04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7720, selfPID=7720, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(406,0xa097b540) malloc: * error for object 0x83d004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(406,0xa097b540) malloc: * error for object 0x103ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(406,0xa097b540) malloc: * error for object 0x806804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(406,0xa097b540) malloc: * error for object 0x806800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(406,0xa097b540) malloc: * error for object 0x802804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(406,0xa097b540) malloc: * error for object 0x802800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(406,0xa097b540) malloc: * error for object 0x802804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(406,0xa097b540) malloc: * error for object 0x802800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(406,0xa097b540) malloc: * error for object 0x802800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x1038604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x83f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7747,0xa097b540) malloc: * error for object 0x81e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:29:58 (392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:58:09 (807): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:58:10 (807): No heartbeat from core client for 30 sec - exiting 20:58:11 (807): No heartbeat from core client for 30 sec - exiting 20:58:12 (807): No heartbeat from core client for 30 sec - exiting 20:58:13 (807): No heartbeat from core client for 30 sec - exiting 20:58:14 (807): No heartbeat from core client for 30 sec - exiting 20:58:15 (807): No heartbeat from core client for 30 sec - exiting 20:58:16 (807): No heartbeat from core client for 30 sec - exiting 20:58:17 (807): No heartbeat from core client for 30 sec - exiting 20:58:18 (807): No heartbeat from core client for 30 sec - exiting 20:58:19 (807): No heartbeat from core client for 30 sec - exiting 20:58:20 (807): No heartbeat from core client for 30 sec - exiting 20:58:21 (807): No heartbeat from core client for 30 sec - exiting 20:58:22 (807): No heartbeat from core client for 30 sec - exiting 20:58:23 (807): No heartbeat from core client for 30 sec - exiting 20:58:24 (807): No heartbeat from core client for 30 sec - exiting 20:58:25 (807): No heartbeat from core client for 30 sec - exiting 20:58:26 (807): No heartbeat from core client for 30 sec - exiting 20:58:27 (807): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135310) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1644, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135310) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1644, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135310) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1644, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135310) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1644, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135310) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1644, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135310) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1644, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 19:35:01	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	803,520	913,798	1.1372
25 Jul 2011 15:58:10	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	777,600	884,595	1.1376
25 Jul 2011 15:00:00	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	751,680	855,112	1.1376
25 Jul 2011 13:32:07	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	725,760	824,631	1.1362
25 Jul 2011 13:32:07	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	699,840	795,089	1.1361
25 Jul 2011 13:32:07	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	673,920	764,831	1.1349
25 Jul 2011 13:32:07	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	648,000	735,579	1.1352
10 Jul 2011 20:08:26	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	622,080	707,162	1.1368
09 Jul 2011 21:37:22	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	596,160	678,128	1.1375
08 Jul 2011 18:02:04	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	570,240	648,646	1.1375
07 Jul 2011 20:08:48	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	544,320	618,563	1.1364
07 Jul 2011 15:37:36	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	518,400	588,861	1.1359
05 Jul 2011 00:19:32	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	492,480	559,397	1.1359
04 Jul 2011 01:48:22	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	466,560	530,350	1.1367
02 Jul 2011 20:12:42	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	440,640	501,346	1.1378
01 Jul 2011 19:36:56	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	414,720	471,977	1.1381
01 Jul 2011 00:50:23	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	388,800	442,927	1.1392
30 Jun 2011 17:36:37	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	362,880	414,045	1.1410
29 Jun 2011 19:49:17	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	336,960	384,523	1.1412
27 Jun 2011 05:53:00	1018716	12926333	hadcm3n_o66p_1940_40_007266495_0	311,040	354,166	1.1387