Task 13929107

Name	hadcm3n_u03s_1980_40_007684219_1
Workunit	7839306
Created	16 Jan 2012, 10:35:38 UTC
Sent	16 Jan 2012, 10:40:50 UTC
Report deadline	16 Apr 2012, 18:08:01 UTC
Received	26 Feb 2012, 1:46:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-2 (0xFFFFFFFE) Unknown error code
Computer ID	500260
Run time	18 days 3 hours 30 min 6 sec
CPU time	15 days 22 hours 15 min 16 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.25 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code -2 (0xfffffffe) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 12:22:31 (13604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:22:32 (13604): No heartbeat from core client for 30 sec - exiting 12:22:33 (13604): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:05:49 (33100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:05:50 (33100): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:55:17 (35928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:55:18 (35928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:56:39 (32220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:43:26 (38512): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 03:43:28 (38512): No heartbeat from core client for 30 sec - exiting 03:43:29 (38512): No heartbeat from core client for 30 sec - exiting 03:43:30 (38512): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:14:58 (35968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:15:00 (35968): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 19:19:56 (39992): No heartbeat from core client for 30 sec - exiting 19:19:57 (39992): No heartbeat from core client for 30 sec - exiting 19:19:58 (39992): No heartbeat from core client for 30 sec - exiting 19:19:59 (39992): No heartbeat from core client for 30 sec - exiting 19:20:00 (39992): No heartbeat from core client for 30 sec - exiting 19:20:01 (39992): No heartbeat from core client for 30 sec - exiting 19:20:02 (39992): No heartbeat from core client for 30 sec - exiting 19:20:03 (39992): No heartbeat from core client for 30 sec - exiting 19:20:04 (39992): No heartbeat from core client for 30 sec - exiting 19:20:05 (39992): No heartbeat from core client for 30 sec - exiting 19:20:06 (39992): No heartbeat from core client for 30 sec - exiting 19:20:07 (39992): No heartbeat from core client for 30 sec - exiting 19:20:08 (39992): No heartbeat from core client for 30 sec - exiting 19:20:09 (39992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:40:17 (38052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:40:19 (38052): No heartbeat from core client for 30 sec - exiting 04:40:20 (38052): No heartbeat from core client for 30 sec - exiting 04:40:21 (38052): No heartbeat from core client for 30 sec - exiting 04:41:07 (36532): No heartbeat from core client for 30 sec - exiting 04:41:08 (36532): No heartbeat from core client for 30 sec - exiting 04:41:09 (36532): No heartbeat from core client for 30 sec - exiting 04:41:10 (36532): No heartbeat from core client for 30 sec - exiting 04:41:11 (36532): No heartbeat from core client for 30 sec - exiting 04:41:12 (36532): No heartbeat from core client for 30 sec - exiting 04:41:13 (36532): No heartbeat from core client for 30 sec - exiting 04:41:14 (36532): No heartbeat from core client for 30 sec - exiting 04:41:15 (36532): No heartbeat from core client for 30 sec - exiting 04:41:16 (36532): No heartbeat from core client for 30 sec - exiting 04:41:17 (36532): No heartbeat from core client for 30 sec - exiting 04:41:18 (36532): No heartbeat from core client for 30 sec - exiting 04:41:19 (36532): No heartbeat from core client for 30 sec - exiting 04:41:20 (36532): No heartbeat from core client for 30 sec - exiting 04:41:21 (36532): No heartbeat from core client for 30 sec - exiting 04:41:22 (36532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:41:23 (36532): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:38:39 (38932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:38:40 (38932): No heartbeat from core client for 30 sec - exiting 04:38:41 (38932): No heartbeat from core client for 30 sec - exiting 04:38:42 (38932): No heartbeat from core client for 30 sec - exiting 04:39:53 (37136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:56:02 (40472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:57:07 (40500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:57:08 (40500): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:12:13 (5964): No heartbeat from core client for 30 sec - exiting 02:12:14 (5964): No heartbeat from core client for 30 sec - exiting 02:12:15 (5964): No heartbeat from core client for 30 sec - exiting 02:12:16 (5964): No heartbeat from core client for 30 sec - exiting 02:12:17 (5964): No heartbeat from core client for 30 sec - exiting 02:12:18 (5964): No heartbeat from core client for 30 sec - exiting 02:12:19 (5964): No heartbeat from core client for 30 sec - exiting 02:12:20 (5964): No heartbeat from core client for 30 sec - exiting 02:12:21 (5964): No heartbeat from core client for 30 sec - exiting 02:12:22 (5964): No heartbeat from core client for 30 sec - exiting 02:12:23 (5964): No heartbeat from core client for 30 sec - exiting 02:12:24 (5964): No heartbeat from core client for 30 sec - exiting 02:12:25 (5964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:29:21 (7512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:29:57 (6112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file D:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_u03s_1980_40_007684219/dataout/ocean_restart.day Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7912, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: error reading file D:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_u03s_1980_40_007684219/dataout/ocean_restart.day Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7912, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: error reading file D:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_u03s_1980_40_007684219/dataout/ocean_restart.day Could not launch model process. Last Error=193 Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Feb 2012 18:17:48	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	777,600	1,354,626	1.7421
25 Feb 2012 11:13:07	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	751,680	1,312,220	1.7457
25 Feb 2012 02:23:43	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	725,760	1,269,592	1.7493
25 Feb 2012 02:23:43	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	699,840	1,227,328	1.7537
15 Feb 2012 05:22:14	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	673,920	1,183,546	1.7562
03 Feb 2012 22:45:52	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	648,000	1,139,835	1.7590
02 Feb 2012 14:03:40	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	622,080	1,096,273	1.7623
02 Feb 2012 14:03:40	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	596,160	1,052,370	1.7652
02 Feb 2012 14:03:40	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	570,240	1,005,696	1.7636
01 Feb 2012 03:29:17	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	544,320	960,208	1.7641
31 Jan 2012 13:07:57	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	518,400	916,504	1.7679
30 Jan 2012 09:13:38	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	492,480	872,163	1.7710
29 Jan 2012 20:55:26	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	466,560	826,908	1.7724
29 Jan 2012 03:06:33	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	440,640	782,185	1.7751
27 Jan 2012 02:26:29	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	414,720	736,267	1.7753
26 Jan 2012 13:12:40	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	388,800	691,447	1.7784
26 Jan 2012 01:34:56	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	362,880	645,567	1.7790
25 Jan 2012 01:59:18	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	336,960	598,678	1.7767
24 Jan 2012 12:12:09	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	311,040	552,196	1.7753
23 Jan 2012 16:32:26	500260	13929107	hadcm3n_u03s_1980_40_007684219_1	285,120	505,983	1.7746