Task 13137202

Name	hadcm3n_yj82_1900_40_007357772_2
Workunit	7555202
Created	10 Jul 2011, 22:05:59 UTC
Sent	10 Jul 2011, 22:24:42 UTC
Report deadline	10 Oct 2011, 5:51:53 UTC
Received	10 Sep 2011, 0:24:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1155482
Run time	15 days 12 hours 3 min 8 sec
CPU time	13 days 0 hours 50 min 13 sec
Validate state	Invalid
Credit	11,819.52
Device peak FLOPS	3.48 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:21:33 (52543): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:44:49 (91576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:52:29 (31986): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 19:38:55 (404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:32:04 (1094): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:32:05 (1094): No heartbeat from core client for 30 sec - exiting 05:32:06 (1094): No heartbeat from core client for 30 sec - exiting 05:32:07 (1094): No heartbeat from core client for 30 sec - exiting 05:32:08 (1094): No heartbeat from core client for 30 sec - exiting 05:32:09 (1094): No heartbeat from core client for 30 sec - exiting 05:32:10 (1094): No heartbeat from core client for 30 sec - exiting 05:32:11 (1094): No heartbeat from core client for 30 sec - exiting 05:32:12 (1094): No heartbeat from core client for 30 sec - exiting 05:32:13 (1094): No heartbeat from core client for 30 sec - exiting 05:32:14 (1094): No heartbeat from core client for 30 sec - exiting 05:32:15 (1094): No heartbeat from core client for 30 sec - exiting 05:32:16 (1094): No heartbeat from core client for 30 sec - exiting 05:32:17 (1094): No heartbeat from core client for 30 sec - exiting 05:32:18 (1094): No heartbeat from core client for 30 sec - exiting 05:32:19 (1094): No heartbeat from core client for 30 sec - exiting 05:32:20 (1094): No heartbeat from core client for 30 sec - exiting 05:32:21 (1094): No heartbeat from core client for 30 sec - exiting 05:32:22 (1094): No heartbeat from core client for 30 sec - exiting 05:32:23 (1094): No heartbeat from core client for 30 sec - exiting 05:32:24 (1094): No heartbeat from core client for 30 sec - exiting 05:32:25 (1094): No heartbeat from core client for 30 sec - exiting 05:32:26 (1094): No heartbeat from core client for 30 sec - exiting 05:32:27 (1094): No heartbeat from core client for 30 sec - exiting 05:32:28 (1094): No heartbeat from core client for 30 sec - exiting 05:32:29 (1094): No heartbeat from core client for 30 sec - exiting 05:32:30 (1094): No heartbeat from core client for 30 sec - exiting 05:32:31 (1094): No heartbeat from core client for 30 sec - exiting 05:32:32 (1094): No heartbeat from core client for 30 sec - exiting 05:32:33 (1094): No heartbeat from core client for 30 sec - exiting 05:32:34 (1094): No heartbeat from core client for 30 sec - exiting 05:32:35 (1094): No heartbeat from core client for 30 sec - exiting 05:32:36 (1094): No heartbeat from core client for 30 sec - exiting 05:32:37 (1094): No heartbeat from core client for 30 sec - exiting 05:32:38 (1094): No heartbeat from core client for 30 sec - exiting 05:32:39 (1094): No heartbeat from core client for 30 sec - exiting 05:32:40 (1094): No heartbeat from core client for 30 sec - exiting 05:32:41 (1094): No heartbeat from core client for 30 sec - exiting 05:32:42 (1094): No heartbeat from core client for 30 sec - exiting 05:32:43 (1094): No heartbeat from core client for 30 sec - exiting 05:32:44 (1094): No heartbeat from core client for 30 sec - exiting 05:32:45 (1094): No heartbeat from core client for 30 sec - exiting 05:32:46 (1094): No heartbeat from core client for 30 sec - exiting 05:32:47 (1094): No heartbeat from core client for 30 sec - exiting 05:32:48 (1094): No heartbeat from core client for 30 sec - exiting 05:32:49 (1094): No heartbeat from core client for 30 sec - exiting 05:32:50 (1094): No heartbeat from core client for 30 sec - exiting 05:32:51 (1094): No heartbeat from core client for 30 sec - exiting 05:32:52 (1094): No heartbeat from core client for 30 sec - exiting 05:32:53 (1094): No heartbeat from core client for 30 sec - exiting 05:32:54 (1094): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:04:59 (16459): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134255) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6503, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134255) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6503, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134255) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6503, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134255) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6503, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134255) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6503, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134255) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6503, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Sep 2011 13:46:23	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	984,960	1,119,162	1.1363
07 Sep 2011 06:39:16	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	959,040	1,088,039	1.1345
06 Sep 2011 07:39:21	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	933,120	1,059,544	1.1355
05 Sep 2011 23:00:00	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	907,200	1,031,286	1.1368
04 Sep 2011 20:49:24	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	881,280	1,003,026	1.1381
04 Sep 2011 12:04:25	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	855,360	974,755	1.1396
04 Sep 2011 03:23:06	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	829,440	946,536	1.1412
03 Sep 2011 14:24:21	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	803,520	918,327	1.1429
03 Sep 2011 04:49:42	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	777,600	890,024	1.1446
31 Aug 2011 07:55:17	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	751,680	861,480	1.1461
30 Aug 2011 22:37:57	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	725,760	830,753	1.1447
30 Aug 2011 13:19:08	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	699,840	800,546	1.1439
30 Aug 2011 04:18:33	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	673,920	770,405	1.1432
29 Aug 2011 19:24:03	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	648,000	740,563	1.1428
29 Aug 2011 10:17:32	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	622,080	710,700	1.1425
29 Aug 2011 01:10:42	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	596,160	680,840	1.1420
28 Aug 2011 15:12:13	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	570,240	649,482	1.1390
28 Aug 2011 04:52:24	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	544,320	618,009	1.1354
27 Aug 2011 19:42:48	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	518,400	587,987	1.1342
27 Aug 2011 10:36:21	1155482	13137202	hadcm3n_yj82_1900_40_007357772_2	492,480	558,133	1.1333