Task 15933758

Name	hadcm3n_4c30_2020_40_008409567_0
Workunit	8560423
Created	22 Aug 2013, 1:55:21 UTC
Sent	23 Aug 2013, 12:50:41 UTC
Report deadline	22 Nov 2013, 20:17:52 UTC
Received	21 Oct 2013, 20:17:36 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1166628
Run time	8 days 22 hours 51 min 22 sec
CPU time	8 days 22 hours 51 min 22 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	2.80 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.4.5</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> 09:55:23 (2552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4656, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4216, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4692, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3544, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4248, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 13:56:19 (4848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:05:11 (4824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5064, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 12:55:31 (4836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:35:16 (2408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:47:29 (4912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:56:36 (4996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:56:37 (4996): No heartbeat from core client for 30 sec - exiting 14:57:23 (3292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:54:15 (4368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:54:17 (4368): No heartbeat from core client for 30 sec - exiting 10:54:18 (4368): No heartbeat from core client for 30 sec - exiting 10:54:19 (4368): No heartbeat from core client for 30 sec - exiting 10:54:20 (4368): No heartbeat from core client for 30 sec - exiting 10:54:21 (4368): No heartbeat from core client for 30 sec - exiting 13:20:58 (4124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:19:41 (3744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:32:14 (2612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4504, iMonCtr=1 Model crash detected, will try to restart... 13:31:41 (1668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:31:42 (1668): No heartbeat from core client for 30 sec - exiting 13:31:43 (1668): No heartbeat from core client for 30 sec - exiting 13:31:44 (1668): No heartbeat from core client for 30 sec - exiting 13:31:45 (1668): No heartbeat from core client for 30 sec - exiting 14:47:03 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:14:19 (2272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:14:20 (2272): No heartbeat from core client for 30 sec - exiting 15:14:21 (2272): No heartbeat from core client for 30 sec - exiting 15:32:33 (4480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:59:41 (5892): No heartbeat from core client for 30 sec - exiting 15:59:42 (5892): No heartbeat from core client for 30 sec - exiting 15:59:43 (5892): No heartbeat from core client for 30 sec - exiting 15:59:44 (5892): No heartbeat from core client for 30 sec - exiting 15:59:45 (5892): No heartbeat from core client for 30 sec - exiting 15:59:46 (5892): No heartbeat from core client for 30 sec - exiting 15:59:47 (5892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4c30ko.pjm8c10 Error converting file to netcdf: dataout/4c30ko.pim8c10 Error converting file to netcdf: dataout/4c30ko.pfm8c10 Error converting file to netcdf: dataout/4c30ka.phm8c10 Error converting file to netcdf: dataout/4c30ka.pgm8c10 Error converting file to netcdf: dataout/4c30ka.pem8c10 Error converting file to netcdf: dataout/4c30ka.pdm8c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:14:21 (5356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:17:23 (5596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:00:02 (3324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:06 (2672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:21:25 (6124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:21:26 (6124): No heartbeat from core client for 30 sec - exiting 13:21:27 (6124): No heartbeat from core client for 30 sec - exiting 13:21:28 (6124): No heartbeat from core client for 30 sec - exiting 13:21:29 (6124): No heartbeat from core client for 30 sec - exiting 13:21:30 (6124): No heartbeat from core client for 30 sec - exiting 13:21:31 (6124): No heartbeat from core client for 30 sec - exiting 13:24:33 (392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:42:50 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:07:11 (5832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:10:14 (5160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:10:15 (5160): No heartbeat from core client for 30 sec - exiting 14:10:16 (5160): No heartbeat from core client for 30 sec - exiting 14:10:17 (5160): No heartbeat from core client for 30 sec - exiting 14:10:18 (5160): No heartbeat from core client for 30 sec - exiting 14:10:19 (5160): No heartbeat from core client for 30 sec - exiting 14:19:25 (2480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:22:28 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:25:57 (2140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:25:58 (2140): No heartbeat from core client for 30 sec - exiting 15:25:59 (2140): No heartbeat from core client for 30 sec - exiting 17:26:34 (308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:24:27 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:27:34 (6104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:02:05 (1188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:02:06 (1188): No heartbeat from core client for 30 sec - exiting 14:02:07 (1188): No heartbeat from core client for 30 sec - exiting 14:20:31 (5920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:26:42 (5964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:26:43 (5964): No heartbeat from core client for 30 sec - exiting 14:26:44 (5964): No heartbeat from core client for 30 sec - exiting 14:26:45 (5964): No heartbeat from core client for 30 sec - exiting 14:26:47 (5964): No heartbeat from core client for 30 sec - exiting 14:26:48 (5964): No heartbeat from core client for 30 sec - exiting 14:41:58 (2420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:01:31 (6808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:35:30 (4420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:21:07 (7008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:32:27 (2796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:30 (6076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:16:58 (5064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:51:23 (6852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:51:24 (6852): No heartbeat from core client for 30 sec - exiting 15:51:25 (6852): No heartbeat from core client for 30 sec - exiting 15:51:26 (6852): No heartbeat from core client for 30 sec - exiting 15:51:27 (6852): No heartbeat from core client for 30 sec - exiting 15:51:28 (6852): No heartbeat from core client for 30 sec - exiting 16:06:35 (7828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:21:46 (8396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:21:47 (8396): No heartbeat from core client for 30 sec - exiting 16:21:48 (8396): No heartbeat from core client for 30 sec - exiting 16:21:49 (8396): No heartbeat from core client for 30 sec - exiting 16:21:50 (8396): No heartbeat from core client for 30 sec - exiting 18:04:35 (1572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:04:36 (1572): No heartbeat from core client for 30 sec - exiting 18:04:37 (1572): No heartbeat from core client for 30 sec - exiting 18:04:38 (1572): No heartbeat from core client for 30 sec - exiting 18:04:39 (1572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Oct 2013 18:24:54	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	492,480	752,959	1.5289
16 Oct 2013 17:40:58	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	466,560	714,597	1.5316
16 Oct 2013 05:08:33	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	440,640	673,163	1.5277
15 Oct 2013 16:37:49	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	414,720	629,787	1.5186
11 Oct 2013 20:58:50	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	388,800	590,961	1.5200
09 Oct 2013 17:16:15	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	362,880	550,334	1.5166
06 Oct 2013 20:16:44	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	336,960	510,016	1.5136
02 Oct 2013 21:10:33	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	311,040	471,986	1.5174
30 Sep 2013 17:42:59	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	285,120	431,578	1.5137
25 Sep 2013 09:22:56	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	259,200	391,769	1.5115
20 Sep 2013 18:31:01	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	233,280	352,429	1.5108
18 Sep 2013 20:03:49	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	207,360	312,683	1.5079
15 Sep 2013 15:00:59	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	181,440	274,263	1.5116
10 Sep 2013 20:12:27	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	155,520	235,333	1.5132
08 Sep 2013 16:44:19	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	129,600	196,610	1.5171
04 Sep 2013 21:40:32	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	103,680	156,988	1.5142
02 Sep 2013 19:39:49	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	77,760	118,323	1.5216
29 Aug 2013 14:57:48	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	51,840	79,319	1.5301
26 Aug 2013 20:06:46	1166628	15933758	hadcm3n_4c30_2020_40_008409567_0	25,920	39,502	1.5240