Task 16083668

Name	hadcm3n_o0cv_2020_40_008375038_4
Workunit	8525897
Created	25 Nov 2013, 22:20:30 UTC
Sent	25 Nov 2013, 23:37:23 UTC
Report deadline	25 Feb 2014, 7:04:34 UTC
Received	21 Dec 2013, 7:03:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	25 (0x00000019) Unknown error code
Computer ID	1158176
Run time	20 days 2 hours 12 min 3 sec
CPU time	18 days 6 hours 11 min 23 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	2.91 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The drive cannot locate a specific area or track on the disk. (0x19) - exit code 25 (0x19) </message> <stderr_txt> 06:46:16 (5292): No heartbeat from core client for 30 sec - exiting 06:46:17 (5292): No heartbeat from core client for 30 sec - exiting 06:46:18 (5292): No heartbeat from core client for 30 sec - exiting 06:46:19 (5292): No heartbeat from core client for 30 sec - exiting 06:46:20 (5292): No heartbeat from core client for 30 sec - exiting 06:46:21 (5292): No heartbeat from core client for 30 sec - exiting 06:46:22 (5292): No heartbeat from core client for 30 sec - exiting 06:46:23 (5292): No heartbeat from core client for 30 sec - exiting 06:46:24 (5292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:47:28 (5796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:48:05 (5980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:48:51 (2288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:49:57 (6640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:18:45 (3692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:32 (7208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:05:16 (3120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:27:53 (6804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:31:44 (15612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:18:23 (9668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:04:01 (29692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:50 (18100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:04 (26368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:07:44 (31448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:25:54 (8252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:14:04 (47780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:22:41 (73000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:35:41 (77916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:38:13 (78640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:15:30 (80680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:17:39 (103412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:17:40 (103412): No heartbeat from core client for 30 sec - exiting 23:17:41 (103412): No heartbeat from core client for 30 sec - exiting 23:17:42 (103412): No heartbeat from core client for 30 sec - exiting 23:17:43 (103412): No heartbeat from core client for 30 sec - exiting 23:17:44 (103412): No heartbeat from core client for 30 sec - exiting 23:17:45 (103412): No heartbeat from core client for 30 sec - exiting 23:17:46 (103412): No heartbeat from core client for 30 sec - exiting 23:17:47 (103412): No heartbeat from core client for 30 sec - exiting 23:17:48 (103412): No heartbeat from core client for 30 sec - exiting 23:17:49 (103412): No heartbeat from core client for 30 sec - exiting 23:21:19 (104840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:25:21 (103824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:25:22 (103824): No heartbeat from core client for 30 sec - exiting 23:25:23 (103824): No heartbeat from core client for 30 sec - exiting 23:25:24 (103824): No heartbeat from core client for 30 sec - exiting 23:25:25 (103824): No heartbeat from core client for 30 sec - exiting 23:25:26 (103824): No heartbeat from core client for 30 sec - exiting 23:25:27 (103824): No heartbeat from core client for 30 sec - exiting 23:25:28 (103824): No heartbeat from core client for 30 sec - exiting 23:25:29 (103824): No heartbeat from core client for 30 sec - exiting 23:25:30 (103824): No heartbeat from core client for 30 sec - exiting 23:25:31 (103824): No heartbeat from core client for 30 sec - exiting 00:05:21 (104380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:07:45 (103976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:09:31 (103648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:09:32 (103648): No heartbeat from core client for 30 sec - exiting 00:09:33 (103648): No heartbeat from core client for 30 sec - exiting 00:09:34 (103648): No heartbeat from core client for 30 sec - exiting 00:09:35 (103648): No heartbeat from core client for 30 sec - exiting 00:09:36 (103648): No heartbeat from core client for 30 sec - exiting 00:09:37 (103648): No heartbeat from core client for 30 sec - exiting 00:09:38 (103648): No heartbeat from core client for 30 sec - exiting 00:09:39 (103648): No heartbeat from core client for 30 sec - exiting 00:09:40 (103648): No heartbeat from core client for 30 sec - exiting 00:09:41 (103648): No heartbeat from core client for 30 sec - exiting 00:29:16 (104980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:31:39 (103528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:55:56 (6088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3608, iMonCtr=1 Model crash detected, will try to restart... 07:11:29 (2764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3028, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3028, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3028, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3028, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3028, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish 17:44:39 (2380): No heartbeat from core client for 30 sec - exiting 17:44:40 (2380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:06 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:35:59 (2300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:32:01 (5576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:35:54 (8440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:45 (4816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:46 (4816): No heartbeat from core client for 30 sec - exiting 07:37:47 (4816): No heartbeat from core client for 30 sec - exiting 07:37:48 (4816): No heartbeat from core client for 30 sec - exiting 07:37:49 (4816): No heartbeat from core client for 30 sec - exiting 07:37:50 (4816): No heartbeat from core client for 30 sec - exiting 07:37:51 (4816): No heartbeat from core client for 30 sec - exiting 07:37:52 (4816): No heartbeat from core client for 30 sec - exiting 07:37:54 (4816): No heartbeat from core client for 30 sec - exiting 07:37:55 (4816): No heartbeat from core client for 30 sec - exiting 07:37:56 (4816): No heartbeat from core client for 30 sec - exiting 07:44:29 (8736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:44:30 (8736): No heartbeat from core client for 30 sec - exiting 07:44:31 (8736): No heartbeat from core client for 30 sec - exiting 07:44:32 (8736): No heartbeat from core client for 30 sec - exiting 07:44:33 (8736): No heartbeat from core client for 30 sec - exiting 07:44:34 (8736): No heartbeat from core client for 30 sec - exiting 07:44:35 (8736): No heartbeat from core client for 30 sec - exiting 07:44:36 (8736): No heartbeat from core client for 30 sec - exiting 07:44:37 (8736): No heartbeat from core client for 30 sec - exiting 07:44:38 (8736): No heartbeat from core client for 30 sec - exiting 07:44:39 (8736): No heartbeat from core client for 30 sec - exiting 07:45:48 (9512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:00 (9240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:01 (9240): No heartbeat from core client for 30 sec - exiting 07:52:02 (9240): No heartbeat from core client for 30 sec - exiting 07:52:03 (9240): No heartbeat from core client for 30 sec - exiting 07:52:04 (9240): No heartbeat from core client for 30 sec - exiting 07:52:05 (9240): No heartbeat from core client for 30 sec - exiting 07:52:06 (9240): No heartbeat from core client for 30 sec - exiting 07:52:07 (9240): No heartbeat from core client for 30 sec - exiting 07:52:08 (9240): No heartbeat from core client for 30 sec - exiting 07:52:10 (9240): No heartbeat from core client for 30 sec - exiting 07:52:11 (9240): No heartbeat from core client for 30 sec - exiting 07:57:53 (4376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:57:54 (4376): No heartbeat from core client for 30 sec - exiting 07:57:55 (4376): No heartbeat from core client for 30 sec - exiting 07:57:56 (4376): No heartbeat from core client for 30 sec - exiting 07:57:57 (4376): No heartbeat from core client for 30 sec - exiting 07:57:58 (4376): No heartbeat from core client for 30 sec - exiting 07:57:59 (4376): No heartbeat from core client for 30 sec - exiting 07:58:01 (4376): No heartbeat from core client for 30 sec - exiting 07:58:02 (4376): No heartbeat from core client for 30 sec - exiting 07:58:03 (4376): No heartbeat from core client for 30 sec - exiting 07:58:04 (4376): No heartbeat from core client for 30 sec - exiting 08:03:57 (6216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:03:58 (6216): No heartbeat from core client for 30 sec - exiting 08:03:59 (6216): No heartbeat from core client for 30 sec - exiting 08:04:00 (6216): No heartbeat from core client for 30 sec - exiting 08:04:01 (6216): No heartbeat from core client for 30 sec - exiting 08:04:02 (6216): No heartbeat from core client for 30 sec - exiting 08:04:03 (6216): No heartbeat from core client for 30 sec - exiting 08:04:04 (6216): No heartbeat from core client for 30 sec - exiting 08:04:05 (6216): No heartbeat from core client for 30 sec - exiting 08:04:07 (6216): No heartbeat from core client for 30 sec - exiting 08:04:08 (6216): No heartbeat from core client for 30 sec - exiting 08:06:54 (5740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:06:55 (5740): No heartbeat from core client for 30 sec - exiting 08:06:56 (5740): No heartbeat from core client for 30 sec - exiting 08:06:57 (5740): No heartbeat from core client for 30 sec - exiting 08:06:58 (5740): No heartbeat from core client for 30 sec - exiting 08:06:59 (5740): No heartbeat from core client for 30 sec - exiting 08:07:00 (5740): No heartbeat from core client for 30 sec - exiting 08:07:01 (5740): No heartbeat from core client for 30 sec - exiting 08:07:02 (5740): No heartbeat from core client for 30 sec - exiting 08:07:03 (5740): No heartbeat from core client for 30 sec - exiting 08:07:05 (5740): No heartbeat from core client for 30 sec - exiting 10:57:31 (5824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:01:09 (2260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:01:10 (2260): No heartbeat from core client for 30 sec - exiting 11:01:11 (2260): No heartbeat from core client for 30 sec - exiting 11:01:12 (2260): No heartbeat from core client for 30 sec - exiting 11:01:13 (2260): No heartbeat from core client for 30 sec - exiting 11:01:15 (2260): No heartbeat from core client for 30 sec - exiting 11:01:16 (2260): No heartbeat from core client for 30 sec - exiting 11:01:17 (2260): No heartbeat from core client for 30 sec - exiting 11:01:18 (2260): No heartbeat from core client for 30 sec - exiting 11:01:19 (2260): No heartbeat from core client for 30 sec - exiting 11:01:20 (2260): No heartbeat from core client for 30 sec - exiting 11:04:13 (4200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:08:02 (3180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:06:58 (1544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:06:59 (1544): No heartbeat from core client for 30 sec - exiting 18:07:55 (18604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:16:57 (19336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:37:00 (16884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:08:59 (10880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:45 (9668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:08:49 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:11:44 (5740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:21 (5212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:13:59 (5592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Dec 2013 05:19:49	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	440,640	1,571,341	3.5660
19 Dec 2013 13:09:39	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	414,720	1,525,967	3.6795
18 Dec 2013 18:56:09	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	388,800	1,474,979	3.7937
04 Dec 2013 20:58:39	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	362,880	691,939	1.9068
04 Dec 2013 06:19:19	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	336,960	643,190	1.9088
03 Dec 2013 16:24:02	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	311,040	594,912	1.9127
03 Dec 2013 02:57:27	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	285,120	546,130	1.9154
02 Dec 2013 11:44:42	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	259,200	498,150	1.9219
01 Dec 2013 20:50:59	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	233,280	445,924	1.9115
01 Dec 2013 05:47:53	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	207,360	395,612	1.9079
30 Nov 2013 14:23:03	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	181,440	343,543	1.8934
29 Nov 2013 23:22:25	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	155,520	293,562	1.8876
29 Nov 2013 08:08:56	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	129,600	241,947	1.8669
28 Nov 2013 17:09:28	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	103,680	192,957	1.8611
28 Nov 2013 02:20:41	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	77,760	147,185	1.8928
27 Nov 2013 10:26:17	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	51,840	98,966	1.9091
26 Nov 2013 18:41:43	1158176	16083668	hadcm3n_o0cv_2020_40_008375038_4	25,920	50,530	1.9495