Task 13782689

Name	hadcm3n_yc9q_1940_40_007616141_1
Workunit	7794271
Created	15 Dec 2011, 14:05:28 UTC
Sent	15 Dec 2011, 14:15:42 UTC
Report deadline	15 Mar 2012, 21:42:53 UTC
Received	28 Dec 2011, 23:05:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1033339
Run time	5 days 12 hours 36 min 57 sec
CPU time	3 days 22 hours 1 min 29 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	3.01 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:24:25 (5825): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:20:15 (28531): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:16 (28531): No heartbeat from core client for 30 sec - exiting 04:21:27 (29051): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:06:48 (21171): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:15:00 (24147): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:49:09 (27563): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:49:27 (27563): No heartbeat from core client for 30 sec - exiting 09:50:45 (27631): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:52 (27657): No heartbeat from core client for 30 sec - exiting 09:51:54 (27657): No heartbeat from core client for 30 sec - exiting 09:51:55 (27657): No heartbeat from core client for 30 sec - exiting 09:51:56 (27657): No heartbeat from core client for 30 sec - exiting 09:51:57 (27657): No heartbeat from core client for 30 sec - exiting 09:51:58 (27657): No heartbeat from core client for 30 sec - exiting 09:51:59 (27657): No heartbeat from core client for 30 sec - exiting 09:52:00 (27657): No heartbeat from core client for 30 sec - exiting 09:52:01 (27657): No heartbeat from core client for 30 sec - exiting 09:52:02 (27657): No heartbeat from core client for 30 sec - exiting 09:52:03 (27657): No heartbeat from core client for 30 sec - exiting 09:52:04 (27657): No heartbeat from core client for 30 sec - exiting 09:52:08 (27657): No heartbeat from core client for 30 sec - exiting 09:52:09 (27657): No heartbeat from core client for 30 sec - exiting 09:52:10 (27657): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:53:52 (27698): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:56:29 (27764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:58:32 (27792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:00:18 (27856): No heartbeat from core client for 30 sec - exiting 10:01:07 (27856): No heartbeat from core client for 30 sec - exiting 10:01:08 (27856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:02:22 (27888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:02:25 (27888): No heartbeat from core client for 30 sec - exiting 10:04:07 (27939): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:05:34 (27994): No heartbeat from core client for 30 sec - exiting 10:05:35 (27994): No heartbeat from core client for 30 sec - exiting 10:05:36 (27994): No heartbeat from core client for 30 sec - exiting 10:05:37 (27994): No heartbeat from core client for 30 sec - exiting 10:05:38 (27994): No heartbeat from core client for 30 sec - exiting 10:05:39 (27994): No heartbeat from core client for 30 sec - exiting 10:05:40 (27994): No heartbeat from core client for 30 sec - exiting 10:05:41 (27994): No heartbeat from core client for 30 sec - exiting 10:05:42 (27994): No heartbeat from core client for 30 sec - exiting 10:05:43 (27994): No heartbeat from core client for 30 sec - exiting 10:05:44 (27994): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:16 (28018): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:19 (28018): No heartbeat from core client for 30 sec - exiting 10:07:20 (28018): No heartbeat from core client for 30 sec - exiting 10:07:21 (28018): No heartbeat from core client for 30 sec - exiting 10:08:30 (28046): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:08:33 (28046): No heartbeat from core client for 30 sec - exiting 10:10:25 (28075): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:15:25 (28122): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:22:48 (28217): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:24:07 (28258): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:24:16 (28258): No heartbeat from core client for 30 sec - exiting 10:24:17 (28258): No heartbeat from core client for 30 sec - exiting 10:32:52 (28298): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:35:11 (28352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:37:13 (28407): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:47:03 (28520): No heartbeat from core client for 30 sec - exiting 10:47:08 (28520): No heartbeat from core client for 30 sec - exiting 10:47:09 (28520): No heartbeat from core client for 30 sec - exiting 10:47:10 (28520): No heartbeat from core client for 30 sec - exiting 10:47:11 (28520): No heartbeat from core client for 30 sec - exiting 10:47:12 (28520): No heartbeat from core client for 30 sec - exiting 10:47:13 (28520): No heartbeat from core client for 30 sec - exiting 10:47:14 (28520): No heartbeat from core client for 30 sec - exiting 10:47:15 (28520): No heartbeat from core client for 30 sec - exiting 10:47:16 (28520): No heartbeat from core client for 30 sec - exiting 10:47:17 (28520): No heartbeat from core client for 30 sec - exiting 10:47:18 (28520): No heartbeat from core client for 30 sec - exiting 10:47:20 (28520): No heartbeat from core client for 30 sec - exiting 10:47:21 (28520): No heartbeat from core client for 30 sec - exiting 10:47:22 (28520): No heartbeat from core client for 30 sec - exiting 10:47:26 (28520): No heartbeat from core client for 30 sec - exiting 10:47:28 (28520): No heartbeat from core client for 30 sec - exiting 10:48:06 (28520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:59 (28520): No heartbeat from core client for 30 sec - exiting 10:49:13 (28520): No heartbeat from core client for 30 sec - exiting 10:58:12 (28592): No heartbeat from core client for 30 sec - exiting 10:58:14 (28592): No heartbeat from core client for 30 sec - exiting 10:58:15 (28592): No heartbeat from core client for 30 sec - exiting 10:58:16 (28592): No heartbeat from core client for 30 sec - exiting 10:58:17 (28592): No heartbeat from core client for 30 sec - exiting 10:58:18 (28592): No heartbeat from core client for 30 sec - exiting 10:58:19 (28592): No heartbeat from core client for 30 sec - exiting 10:58:20 (28592): No heartbeat from core client for 30 sec - exiting 10:58:21 (28592): No heartbeat from core client for 30 sec - exiting 10:58:22 (28592): No heartbeat from core client for 30 sec - exiting 10:58:23 (28592): No heartbeat from core client for 30 sec - exiting 10:58:24 (28592): No heartbeat from core client for 30 sec - exiting 10:58:25 (28592): No heartbeat from core client for 30 sec - exiting 10:58:27 (28592): No heartbeat from core client for 30 sec - exiting 10:58:28 (28592): No heartbeat from core client for 30 sec - exiting 10:58:29 (28592): No heartbeat from core client for 30 sec - exiting 10:58:30 (28592): No heartbeat from core client for 30 sec - exiting 10:58:31 (28592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:00:30 (28646): No heartbeat from core client for 30 sec - exiting 11:01:06 (28646): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:02:14 (28663): No heartbeat from core client for 30 sec - exiting 11:02:17 (28663): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:04:11 (28696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28753, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28753, iMonCtr=1 Model crash detected, will try to restart... 11:54:45 (28753): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28908, iMonCtr=1 Model crash detected, will try to restart... 15:51:29 (28908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:57:44 (29000): No heartbeat from core client for 30 sec - exiting 15:57:51 (29000): No heartbeat from core client for 30 sec - exiting 15:57:53 (29000): No heartbeat from core client for 30 sec - exiting 15:57:54 (29000): No heartbeat from core client for 30 sec - exiting 15:57:55 (29000): No heartbeat from core client for 30 sec - exiting 15:57:56 (29000): No heartbeat from core client for 30 sec - exiting 15:57:57 (29000): No heartbeat from core client for 30 sec - exiting 15:57:58 (29000): No heartbeat from core client for 30 sec - exiting 15:57:59 (29000): No heartbeat from core client for 30 sec - exiting 15:58:00 (29000): No heartbeat from core client for 30 sec - exiting 15:58:01 (29000): No heartbeat from core client for 30 sec - exiting 15:58:02 (29000): No heartbeat from core client for 30 sec - exiting 15:58:04 (29000): No heartbeat from core client for 30 sec - exiting 15:58:05 (29000): No heartbeat from core client for 30 sec - exiting 15:58:06 (29000): No heartbeat from core client for 30 sec - exiting 15:58:07 (29000): No heartbeat from core client for 30 sec - exiting 15:58:08 (29000): No heartbeat from core client for 30 sec - exiting 15:58:09 (29000): No heartbeat from core client for 30 sec - exiting 15:58:10 (29000): No heartbeat from core client for 30 sec - exiting 15:58:11 (29000): No heartbeat from core client for 30 sec - exiting 15:58:12 (29000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:00:10 (29058): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:05:18 (29096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:16:23 (29164): No heartbeat from core client for 30 sec - exiting 16:16:58 (29164): No heartbeat from core client for 30 sec - exiting 16:17:00 (29164): No heartbeat from core client for 30 sec - exiting 16:17:04 (29164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:11:55 (29245): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29294, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29294, iMonCtr=1 Model crash detected, will try to restart... 21:34:45 (29294): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29475, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( 00:25:19 (29475): No heartbeat from core client for 30 sec - exiting Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Dec 2011 23:04:52	1033339	13782689	hadcm3n_yc9q_1940_40_007616141_1	181,440	300,390	1.6556
24 Dec 2011 15:51:16	1033339	13782689	hadcm3n_yc9q_1940_40_007616141_1	155,520	257,253	1.6541
22 Dec 2011 19:25:59	1033339	13782689	hadcm3n_yc9q_1940_40_007616141_1	129,600	213,625	1.6483
21 Dec 2011 12:08:44	1033339	13782689	hadcm3n_yc9q_1940_40_007616141_1	103,680	170,736	1.6468
20 Dec 2011 05:57:10	1033339	13782689	hadcm3n_yc9q_1940_40_007616141_1	77,760	128,175	1.6483
19 Dec 2011 02:01:10	1033339	13782689	hadcm3n_yc9q_1940_40_007616141_1	51,840	85,749	1.6541
17 Dec 2011 15:25:22	1033339	13782689	hadcm3n_yc9q_1940_40_007616141_1	25,920	43,040	1.6605