Task 13663683

Name	hadcm3n_o136_1980_40_007425808_2
Workunit	7623311
Created	26 Nov 2011, 14:01:44 UTC
Sent	26 Nov 2011, 14:02:06 UTC
Report deadline	25 Feb 2012, 21:29:17 UTC
Received	12 Dec 2011, 13:35:31 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	726409
Run time	6 days 19 hours 20 min 9 sec
CPU time	6 days 18 hours 50 min 16 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	2.99 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> Za?izeni nezna tento p?ikaz. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7400, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6800, selfPID=6800, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o136ko.pji6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:31:39 (4568): No heartbeat from core client for 30 sec - exiting 11:31:40 (4568): No heartbeat from core client for 30 sec - exiting 11:31:41 (4568): No heartbeat from core client for 30 sec - exiting 11:31:42 (4568): No heartbeat from core client for 30 sec - exiting 11:31:43 (4568): No heartbeat from core client for 30 sec - exiting 11:31:44 (4568): No heartbeat from core client for 30 sec - exiting 11:31:45 (4568): No heartbeat from core client for 30 sec - exiting 11:31:46 (4568): No heartbeat from core client for 30 sec - exiting 11:31:48 (4568): No heartbeat from core client for 30 sec - exiting 11:31:49 (4568): No heartbeat from core client for 30 sec - exiting 11:31:50 (4568): No heartbeat from core client for 30 sec - exiting 11:31:51 (4568): No heartbeat from core client for 30 sec - exiting 11:31:52 (4568): No heartbeat from core client for 30 sec - exiting 11:31:53 (4568): No heartbeat from core client for 30 sec - exiting 11:31:54 (4568): No heartbeat from core client for 30 sec - exiting 11:31:55 (4568): No heartbeat from core client for 30 sec - exiting 11:31:56 (4568): No heartbeat from core client for 30 sec - exiting 11:31:57 (4568): No heartbeat from core client for 30 sec - exiting 11:31:58 (4568): No heartbeat from core client for 30 sec - exiting 11:32:00 (4568): No heartbeat from core client for 30 sec - exiting 11:32:01 (4568): No heartbeat from core client for 30 sec - exiting 11:32:02 (4568): No heartbeat from core client for 30 sec - exiting 11:32:03 (4568): No heartbeat from core client for 30 sec - exiting 11:32:04 (4568): No heartbeat from core client for 30 sec - exiting 11:32:05 (4568): No heartbeat from core client for 30 sec - exiting 11:32:06 (4568): No heartbeat from core client for 30 sec - exiting 11:32:07 (4568): No heartbeat from core client for 30 sec - exiting 11:32:08 (4568): No heartbeat from core client for 30 sec - exiting 11:32:09 (4568): No heartbeat from core client for 30 sec - exiting 11:32:10 (4568): No heartbeat from core client for 30 sec - exiting 11:32:11 (4568): No heartbeat from core client for 30 sec - exiting 11:32:12 (4568): No heartbeat from core client for 30 sec - exiting 11:32:13 (4568): No heartbeat from core client for 30 sec - exiting 11:32:14 (4568): No heartbeat from core client for 30 sec - exiting 11:32:15 (4568): No heartbeat from core client for 30 sec - exiting 11:32:17 (4568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4124, iMonCtr=1 Model crash detected, will try to restart... 16:02:37 (5052): No heartbeat from core client for 30 sec - exiting 16:02:39 (5052): No heartbeat from core client for 30 sec - exiting 16:02:40 (5052): No heartbeat from core client for 30 sec - exiting 16:02:41 (5052): No heartbeat from core client for 30 sec - exiting 16:02:42 (5052): No heartbeat from core client for 30 sec - exiting 16:02:43 (5052): No heartbeat from core client for 30 sec - exiting 16:02:44 (5052): No heartbeat from core client for 30 sec - exiting 16:02:45 (5052): No heartbeat from core client for 30 sec - exiting 16:02:46 (5052): No heartbeat from core client for 30 sec - exiting 16:02:47 (5052): No heartbeat from core client for 30 sec - exiting 16:02:48 (5052): No heartbeat from core client for 30 sec - exiting 16:02:50 (5052): No heartbeat from core client for 30 sec - exiting 16:02:51 (5052): No heartbeat from core client for 30 sec - exiting 16:02:52 (5052): No heartbeat from core client for 30 sec - exiting 16:02:53 (5052): No heartbeat from core client for 30 sec - exiting 16:02:54 (5052): No heartbeat from core client for 30 sec - exiting 16:02:55 (5052): No heartbeat from core client for 30 sec - exiting 16:02:56 (5052): No heartbeat from core client for 30 sec - exiting 16:02:57 (5052): No heartbeat from core client for 30 sec - exiting 16:02:58 (5052): No heartbeat from core client for 30 sec - exiting 16:02:59 (5052): No heartbeat from core client for 30 sec - exiting 16:03:00 (5052): No heartbeat from core client for 30 sec - exiting 16:03:02 (5052): No heartbeat from core client for 30 sec - exiting 16:03:03 (5052): No heartbeat from core client for 30 sec - exiting 16:03:04 (5052): No heartbeat from core client for 30 sec - exiting 16:03:05 (5052): No heartbeat from core client for 30 sec - exiting 16:03:06 (5052): No heartbeat from core client for 30 sec - exiting 16:03:07 (5052): No heartbeat from core client for 30 sec - exiting 16:03:08 (5052): No heartbeat from core client for 30 sec - exiting 16:03:09 (5052): No heartbeat from core client for 30 sec - exiting 16:03:10 (5052): No heartbeat from core client for 30 sec - exiting 16:03:11 (5052): No heartbeat from core client for 30 sec - exiting 16:03:13 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:43:51 (5352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:48 (4844): No heartbeat from core client for 30 sec - exiting 13:24:50 (4844): No heartbeat from core client for 30 sec - exiting 13:24:51 (4844): No heartbeat from core client for 30 sec - exiting 13:24:52 (4844): No heartbeat from core client for 30 sec - exiting 13:24:53 (4844): No heartbeat from core client for 30 sec - exiting 13:24:54 (4844): No heartbeat from core client for 30 sec - exiting 13:24:55 (4844): No heartbeat from core client for 30 sec - exiting 13:24:56 (4844): No heartbeat from core client for 30 sec - exiting 13:24:57 (4844): No heartbeat from core client for 30 sec - exiting 13:24:58 (4844): No heartbeat from core client for 30 sec - exiting 13:24:59 (4844): No heartbeat from core client for 30 sec - exiting 13:25:00 (4844): No heartbeat from core client for 30 sec - exiting 13:25:02 (4844): No heartbeat from core client for 30 sec - exiting 13:25:03 (4844): No heartbeat from core client for 30 sec - exiting 13:25:04 (4844): No heartbeat from core client for 30 sec - exiting 13:25:05 (4844): No heartbeat from core client for 30 sec - exiting 13:25:06 (4844): No heartbeat from core client for 30 sec - exiting 13:25:07 (4844): No heartbeat from core client for 30 sec - exiting 13:25:08 (4844): No heartbeat from core client for 30 sec - exiting 13:25:09 (4844): No heartbeat from core client for 30 sec - exiting 13:25:10 (4844): No heartbeat from core client for 30 sec - exiting 13:25:11 (4844): No heartbeat from core client for 30 sec - exiting 13:25:12 (4844): No heartbeat from core client for 30 sec - exiting 13:25:14 (4844): No heartbeat from core client for 30 sec - exiting 13:25:15 (4844): No heartbeat from core client for 30 sec - exiting 13:25:16 (4844): No heartbeat from core client for 30 sec - exiting 13:25:17 (4844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:45:49 (4964): No heartbeat from core client for 30 sec - exiting 18:45:50 (4964): No heartbeat from core client for 30 sec - exiting 18:45:51 (4964): No heartbeat from core client for 30 sec - exiting 18:45:52 (4964): No heartbeat from core client for 30 sec - exiting 18:45:53 (4964): No heartbeat from core client for 30 sec - exiting 18:45:54 (4964): No heartbeat from core client for 30 sec - exiting 18:45:55 (4964): No heartbeat from core client for 30 sec - exiting 18:45:56 (4964): No heartbeat from core client for 30 sec - exiting 18:45:58 (4964): No heartbeat from core client for 30 sec - exiting 18:45:59 (4964): No heartbeat from core client for 30 sec - exiting 18:46:00 (4964): No heartbeat from core client for 30 sec - exiting 18:46:01 (4964): No heartbeat from core client for 30 sec - exiting 18:46:02 (4964): No heartbeat from core client for 30 sec - exiting 18:46:03 (4964): No heartbeat from core client for 30 sec - exiting 18:46:04 (4964): No heartbeat from core client for 30 sec - exiting 18:46:05 (4964): No heartbeat from core client for 30 sec - exiting 18:46:06 (4964): No heartbeat from core client for 30 sec - exiting 18:46:07 (4964): No heartbeat from core client for 30 sec - exiting 18:46:08 (4964): No heartbeat from core client for 30 sec - exiting 18:46:10 (4964): No heartbeat from core client for 30 sec - exiting 18:46:11 (4964): No heartbeat from core client for 30 sec - exiting 18:46:12 (4964): No heartbeat from core client for 30 sec - exiting 18:46:13 (4964): No heartbeat from core client for 30 sec - exiting 18:46:14 (4964): No heartbeat from core client for 30 sec - exiting 18:46:15 (4964): No heartbeat from core client for 30 sec - exiting 18:46:16 (4964): No heartbeat from core client for 30 sec - exiting 18:46:17 (4964): No heartbeat from core client for 30 sec - exiting 18:46:18 (4964): No heartbeat from core client for 30 sec - exiting 18:46:19 (4964): No heartbeat from core client for 30 sec - exiting 18:46:20 (4964): No heartbeat from core client for 30 sec - exiting 18:46:21 (4964): No heartbeat from core client for 30 sec - exiting 18:46:22 (4964): No heartbeat from core client for 30 sec - exiting 18:46:23 (4964): No heartbeat from core client for 30 sec - exiting 18:46:24 (4964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4968, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4968, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4608, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 00:49:36 (2728): No heartbeat from core client for 30 sec - exiting 00:49:37 (2728): No heartbeat from core client for 30 sec - exiting 00:49:38 (2728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4932, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3808, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3808, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Dec 2011 14:33:16	726409	13663683	hadcm3n_o136_1980_40_007425808_2	440,640	574,880	1.3046
10 Dec 2011 17:33:10	726409	13663683	hadcm3n_o136_1980_40_007425808_2	414,720	539,745	1.3015
08 Dec 2011 19:28:42	726409	13663683	hadcm3n_o136_1980_40_007425808_2	388,800	505,550	1.3003
08 Dec 2011 09:58:33	726409	13663683	hadcm3n_o136_1980_40_007425808_2	362,880	471,573	1.2995
07 Dec 2011 16:39:37	726409	13663683	hadcm3n_o136_1980_40_007425808_2	336,960	438,399	1.3010
06 Dec 2011 23:47:39	726409	13663683	hadcm3n_o136_1980_40_007425808_2	311,040	405,077	1.3023
06 Dec 2011 14:21:37	726409	13663683	hadcm3n_o136_1980_40_007425808_2	285,120	371,237	1.3020
05 Dec 2011 20:20:10	726409	13663683	hadcm3n_o136_1980_40_007425808_2	259,200	337,172	1.3008
05 Dec 2011 11:11:34	726409	13663683	hadcm3n_o136_1980_40_007425808_2	233,280	304,195	1.3040
04 Dec 2011 13:36:03	726409	13663683	hadcm3n_o136_1980_40_007425808_2	207,360	268,342	1.2941
03 Dec 2011 11:51:57	726409	13663683	hadcm3n_o136_1980_40_007425808_2	181,440	235,315	1.2969
01 Dec 2011 21:43:54	726409	13663683	hadcm3n_o136_1980_40_007425808_2	155,520	201,170	1.2935
30 Nov 2011 08:51:37	726409	13663683	hadcm3n_o136_1980_40_007425808_2	129,600	165,026	1.2733
29 Nov 2011 14:32:17	726409	13663683	hadcm3n_o136_1980_40_007425808_2	103,680	132,713	1.2800
28 Nov 2011 20:26:24	726409	13663683	hadcm3n_o136_1980_40_007425808_2	77,760	97,857	1.2584
28 Nov 2011 10:06:00	726409	13663683	hadcm3n_o136_1980_40_007425808_2	51,840	66,517	1.2831
27 Nov 2011 14:37:14	726409	13663683	hadcm3n_o136_1980_40_007425808_2	25,920	34,598	1.3348