Task 13291775

Name	hadcm3n_p7a5_1940_40_007421669_1
Workunit	7619304
Created	25 Aug 2011, 3:03:16 UTC
Sent	25 Aug 2011, 3:09:57 UTC
Report deadline	24 Nov 2011, 10:37:08 UTC
Received	4 Nov 2011, 17:52:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1149499
Run time	18 days 19 hours 1 min 49 sec
CPU time	15 days 18 hours 35 min 53 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.99 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6892, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:49:50 (5932): No heartbeat from core client for 30 sec - exiting 10:49:52 (5932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Ocean Restart file copy failed on p7a5ko.dae45o0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:29:33 (4488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:02:43 (6688): No heartbeat from core client for 30 sec - exiting 12:02:44 (6688): No heartbeat from core client for 30 sec - exiting 12:02:45 (6688): No heartbeat from core client for 30 sec - exiting 12:02:46 (6688): No heartbeat from core client for 30 sec - exiting 12:02:47 (6688): No heartbeat from core client for 30 sec - exiting 12:02:48 (6688): No heartbeat from core client for 30 sec - exiting 12:02:49 (6688): No heartbeat from core client for 30 sec - exiting 12:02:50 (6688): No heartbeat from core client for 30 sec - exiting 12:02:51 (6688): No heartbeat from core client for 30 sec - exiting 12:02:52 (6688): No heartbeat from core client for 30 sec - exiting 12:02:53 (6688): No heartbeat from core client for 30 sec - exiting 12:02:54 (6688): No heartbeat from core client for 30 sec - exiting 12:02:55 (6688): No heartbeat from core client for 30 sec - exiting 12:02:56 (6688): No heartbeat from core client for 30 sec - exiting 12:02:57 (6688): No heartbeat from core client for 30 sec - exiting 12:02:58 (6688): No heartbeat from core client for 30 sec - exiting 12:02:59 (6688): No heartbeat from core client for 30 sec - exiting 12:03:00 (6688): No heartbeat from core client for 30 sec - exiting 12:03:01 (6688): No heartbeat from core client for 30 sec - exiting 12:03:02 (6688): No heartbeat from core client for 30 sec - exiting 12:03:03 (6688): No heartbeat from core client for 30 sec - exiting 12:03:04 (6688): No heartbeat from core client for 30 sec - exiting 12:03:05 (6688): No heartbeat from core client for 30 sec - exiting 12:03:06 (6688): No heartbeat from core client for 30 sec - exiting 12:03:07 (6688): No heartbeat from core client for 30 sec - exiting 12:03:08 (6688): No heartbeat from core client for 30 sec - exiting 12:03:10 (6688): No heartbeat from core client for 30 sec - exiting 12:03:11 (6688): No heartbeat from core client for 30 sec - exiting 12:03:12 (6688): No heartbeat from core client for 30 sec - exiting 12:03:13 (6688): No heartbeat from core client for 30 sec - exiting 12:03:14 (6688): No heartbeat from core client for 30 sec - exiting 12:03:15 (6688): No heartbeat from core client for 30 sec - exiting 12:03:16 (6688): No heartbeat from core client for 30 sec - exiting 12:03:17 (6688): No heartbeat from core client for 30 sec - exiting 12:03:18 (6688): No heartbeat from core client for 30 sec - exiting 12:03:19 (6688): No heartbeat from core client for 30 sec - exiting 12:03:20 (6688): No heartbeat from core client for 30 sec - exiting 12:03:21 (6688): No heartbeat from core client for 30 sec - exiting 12:03:22 (6688): No heartbeat from core client for 30 sec - exiting 12:03:23 (6688): No heartbeat from core client for 30 sec - exiting 12:03:24 (6688): No heartbeat from core client for 30 sec - exiting 12:03:25 (6688): No heartbeat from core client for 30 sec - exiting 12:03:26 (6688): No heartbeat from core client for 30 sec - exiting 12:03:27 (6688): No heartbeat from core client for 30 sec - exiting 12:03:28 (6688): No heartbeat from core client for 30 sec - exiting 12:03:29 (6688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:03:30 (6688): No heartbeat from core client for 30 sec - exiting 12:03:31 (6688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:54:17 (3284): No heartbeat from core client for 30 sec - exiting 19:54:18 (3284): No heartbeat from core client for 30 sec - exiting 19:54:19 (3284): No heartbeat from core client for 30 sec - exiting 19:54:20 (3284): No heartbeat from core client for 30 sec - exiting 19:54:21 (3284): No heartbeat from core client for 30 sec - exiting 19:54:22 (3284): No heartbeat from core client for 30 sec - exiting 19:54:23 (3284): No heartbeat from core client for 30 sec - exiting 19:54:24 (3284): No heartbeat from core client for 30 sec - exiting 19:54:25 (3284): No heartbeat from core client for 30 sec - exiting 19:54:26 (3284): No heartbeat from core client for 30 sec - exiting 19:54:27 (3284): No heartbeat from core client for 30 sec - exiting 19:54:28 (3284): No heartbeat from core client for 30 sec - exiting 19:54:30 (3284): No heartbeat from core client for 30 sec - exiting 19:54:31 (3284): No heartbeat from core client for 30 sec - exiting 19:54:32 (3284): No heartbeat from core client for 30 sec - exiting 19:54:33 (3284): No heartbeat from core client for 30 sec - exiting 19:54:34 (3284): No heartbeat from core client for 30 sec - exiting 19:54:35 (3284): No heartbeat from core client for 30 sec - exiting 19:54:36 (3284): No heartbeat from core client for 30 sec - exiting 19:54:37 (3284): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3228, selfPID=3228, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on p7a5ko.daf34h0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on p7a5ko.daf57b0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:55:15 (5404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:00:19 (3528): No heartbeat from core client for 30 sec - exiting 10:00:20 (3528): No heartbeat from core client for 30 sec - exiting 10:00:22 (3528): No heartbeat from core client for 30 sec - exiting 10:00:23 (3528): No heartbeat from core client for 30 sec - exiting 10:00:24 (3528): No heartbeat from core client for 30 sec - exiting 10:00:25 (3528): No heartbeat from core client for 30 sec - exiting 10:00:26 (3528): No heartbeat from core client for 30 sec - exiting 10:00:27 (3528): No heartbeat from core client for 30 sec - exiting 10:00:28 (3528): No heartbeat from core client for 30 sec - exiting 10:00:29 (3528): No heartbeat from core client for 30 sec - exiting 10:00:30 (3528): No heartbeat from core client for 30 sec - exiting 10:00:31 (3528): No heartbeat from core client for 30 sec - exiting 10:00:32 (3528): No heartbeat from core client for 30 sec - exiting 10:00:33 (3528): No heartbeat from core client for 30 sec - exiting 10:00:34 (3528): No heartbeat from core client for 30 sec - exiting 10:00:35 (3528): No heartbeat from core client for 30 sec - exiting 10:00:36 (3528): No heartbeat from core client for 30 sec - exiting 10:00:37 (3528): No heartbeat from core client for 30 sec - exiting 10:00:38 (3528): No heartbeat from core client for 30 sec - exiting 10:00:39 (3528): No heartbeat from core client for 30 sec - exiting 10:00:40 (3528): No heartbeat from core client for 30 sec - exiting 10:00:41 (3528): No heartbeat from core client for 30 sec - exiting 10:00:42 (3528): No heartbeat from core client for 30 sec - exiting 10:00:43 (3528): No heartbeat from core client for 30 sec - exiting 10:00:44 (3528): No heartbeat from core client for 30 sec - exiting 10:00:45 (3528): No heartbeat from core client for 30 sec - exiting 10:00:46 (3528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on p7a5ko.dag2890 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on p7a5ko.dag4bb0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on p7a5ko.dag71e0 Ocean Restart file copy failed on p7a5ko.dag71f0 CPDN Monitor - Quit request from BOINC... 14:40:59 (6264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:41:00 (6264): No heartbeat from core client for 30 sec - exiting 14:42:57 (3408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on p7a5ko.dag92o0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:35:45 (4552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:35:47 (4552): No heartbeat from core client for 30 sec - exiting 11:35:48 (4552): No heartbeat from core client for 30 sec - exiting 11:37:05 (4480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4316, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on p7a5ko.dah3c50 Ocean Restart file copy failed on p7a5ko.dah3c60 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1380, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... 11:30:31 (4480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:34:05 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on p7a5ko.dah8bi0 Ocean Restart file copy failed on p7a5ko.dah8bj0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on p7a5ko.dah9460 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6128, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Nov 2011 20:38:25	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	1,036,800	1,362,948	1.3146
02 Nov 2011 14:33:22	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	1,010,880	1,329,168	1.3149
31 Oct 2011 17:24:10	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	984,960	1,294,762	1.3145
31 Oct 2011 17:13:50	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	959,040	1,261,117	1.3150
31 Oct 2011 13:15:01	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	933,120	1,227,642	1.3156
31 Oct 2011 13:15:01	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	907,200	1,194,914	1.3171
31 Oct 2011 13:15:00	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	881,280	1,161,766	1.3183
31 Oct 2011 13:15:00	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	855,360	1,128,277	1.3191
17 Oct 2011 11:08:45	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	829,440	1,094,665	1.3198
17 Oct 2011 11:08:45	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	803,520	1,060,608	1.3200
17 Oct 2011 11:08:45	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	777,600	1,026,744	1.3204
11 Oct 2011 11:37:40	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	751,680	993,036	1.3211
10 Oct 2011 23:32:33	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	725,760	959,669	1.3223
10 Oct 2011 11:58:37	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	699,840	925,971	1.3231
10 Oct 2011 01:01:45	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	673,920	893,067	1.3252
09 Oct 2011 13:56:06	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	648,000	859,846	1.3269
09 Oct 2011 03:11:25	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	622,080	827,134	1.3296
08 Oct 2011 15:52:04	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	596,160	793,472	1.3310
08 Oct 2011 05:23:56	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	570,240	760,186	1.3331
07 Oct 2011 19:13:16	1149499	13291775	hadcm3n_p7a5_1940_40_007421669_1	544,320	726,951	1.3355