Task 13319242

Name	hadcm3n_t05n_1940_40_007432912_2
Workunit	7630415
Created	31 Aug 2011, 21:48:14 UTC
Sent	31 Aug 2011, 21:57:13 UTC
Report deadline	1 Dec 2011, 5:24:24 UTC
Received	6 Oct 2011, 5:00:15 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	991725
Run time	15 days 22 hours 52 min 31 sec
CPU time	13 days 17 hours 15 min 23 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.95 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:14:37 (14153): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:09:12 (21830): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:09:13 (21830): No heartbeat from core client for 30 sec - exiting 01:09:14 (21830): No heartbeat from core client for 30 sec - exiting 01:09:15 (21830): No heartbeat from core client for 30 sec - exiting 01:09:16 (21830): No heartbeat from core client for 30 sec - exiting 01:09:17 (21830): No heartbeat from core client for 30 sec - exiting 01:09:18 (21830): No heartbeat from core client for 30 sec - exiting 01:09:19 (21830): No heartbeat from core client for 30 sec - exiting 01:09:20 (21830): No heartbeat from core client for 30 sec - exiting 01:09:21 (21830): No heartbeat from core client for 30 sec - exiting 01:09:22 (21830): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:14:10 (31092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:14:12 (31092): No heartbeat from core client for 30 sec - exiting 06:14:13 (31092): No heartbeat from core client for 30 sec - exiting 06:14:23 (31092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 10:48:47 (34643): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:48 (34643): No heartbeat from core client for 30 sec - exiting 10:48:49 (34643): No heartbeat from core client for 30 sec - exiting 10:48:50 (34643): No heartbeat from core client for 30 sec - exiting 10:48:51 (34643): No heartbeat from core client for 30 sec - exiting 10:48:52 (34643): No heartbeat from core client for 30 sec - exiting 10:48:53 (34643): No heartbeat from core client for 30 sec - exiting 10:48:54 (34643): No heartbeat from core client for 30 sec - exiting 10:48:55 (34643): No heartbeat from core client for 30 sec - exiting 10:48:56 (34643): No heartbeat from core client for 30 sec - exiting 10:48:57 (34643): No heartbeat from core client for 30 sec - exiting 10:48:58 (34643): No heartbeat from core client for 30 sec - exiting 10:48:59 (34643): No heartbeat from core client for 30 sec - exiting 10:49:00 (34643): No heartbeat from core client for 30 sec - exiting 10:49:01 (34643): No heartbeat from core client for 30 sec - exiting 10:49:02 (34643): No heartbeat from core client for 30 sec - exiting 10:49:03 (34643): No heartbeat from core client for 30 sec - exiting 10:49:04 (34643): No heartbeat from core client for 30 sec - exiting 10:49:05 (34643): No heartbeat from core client for 30 sec - exiting 10:49:06 (34643): No heartbeat from core client for 30 sec - exiting 10:49:07 (34643): No heartbeat from core client for 30 sec - exiting 10:49:08 (34643): No heartbeat from core client for 30 sec - exiting 10:49:09 (34643): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=242, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133545) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1903, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133545) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1903, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133545) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1903, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133545) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1903, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133545) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1903, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Oct 2011 16:13:49	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	777,600	1,156,971	1.4879
02 Oct 2011 02:51:05	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	751,680	1,118,976	1.4886
01 Oct 2011 12:11:24	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	725,760	1,079,490	1.4874
30 Sep 2011 08:56:23	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	699,840	1,039,512	1.4854
29 Sep 2011 20:34:23	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	673,920	1,001,037	1.4854
28 Sep 2011 11:31:31	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	648,000	961,309	1.4835
27 Sep 2011 23:22:28	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	622,080	923,270	1.4842
26 Sep 2011 22:20:41	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	596,160	884,063	1.4829
25 Sep 2011 11:42:31	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	570,240	845,160	1.4821
23 Sep 2011 15:35:55	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	544,320	806,992	1.4826
22 Sep 2011 10:13:59	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	518,400	766,185	1.4780
21 Sep 2011 11:13:04	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	492,480	726,849	1.4759
20 Sep 2011 16:43:05	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	466,560	688,300	1.4753
19 Sep 2011 13:06:03	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	440,640	648,710	1.4722
17 Sep 2011 23:24:04	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	414,720	609,741	1.4702
16 Sep 2011 16:18:44	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	388,800	570,901	1.4684
15 Sep 2011 10:40:30	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	362,880	532,431	1.4672
14 Sep 2011 05:37:37	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	336,960	492,287	1.4610
13 Sep 2011 03:01:40	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	311,040	453,237	1.4572
11 Sep 2011 19:40:55	991725	13319242	hadcm3n_t05n_1940_40_007432912_2	285,120	415,002	1.4555