Task 12918767

Name	hadcm3n_o048_1940_40_007264533_0
Workunit	7462773
Created	1 Jun 2011, 19:33:01 UTC
Sent	1 Jun 2011, 19:33:04 UTC
Report deadline	1 Sep 2011, 3:00:15 UTC
Received	18 Jul 2011, 14:27:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1036099
Run time	6 days 0 hours 18 min 9 sec
CPU time	5 days 10 hours 42 min 59 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.62 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 06:51:23 (3212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:44:14 (5540): No heartbeat from core client for 30 sec - exiting 07:44:15 (5540): No heartbeat from core client for 30 sec - exiting 07:44:16 (5540): No heartbeat from core client for 30 sec - exiting 07:44:17 (5540): No heartbeat from core client for 30 sec - exiting 07:44:18 (5540): No heartbeat from core client for 30 sec - exiting 07:44:19 (5540): No heartbeat from core client for 30 sec - exiting 07:44:20 (5540): No heartbeat from core client for 30 sec - exiting 07:44:21 (5540): No heartbeat from core client for 30 sec - exiting 07:44:22 (5540): No heartbeat from core client for 30 sec - exiting 07:44:23 (5540): No heartbeat from core client for 30 sec - exiting 07:44:24 (5540): No heartbeat from core client for 30 sec - exiting 07:44:25 (5540): No heartbeat from core client for 30 sec - exiting 07:44:26 (5540): No heartbeat from core client for 30 sec - exiting 07:44:27 (5540): No heartbeat from core client for 30 sec - exiting 07:44:28 (5540): No heartbeat from core client for 30 sec - exiting 07:44:29 (5540): No heartbeat from core client for 30 sec - exiting 07:44:30 (5540): No heartbeat from core client for 30 sec - exiting 07:44:31 (5540): No heartbeat from core client for 30 sec - exiting 07:44:32 (5540): No heartbeat from core client for 30 sec - exiting 07:44:33 (5540): No heartbeat from core client for 30 sec - exiting 07:44:34 (5540): No heartbeat from core client for 30 sec - exiting 07:44:35 (5540): No heartbeat from core client for 30 sec - exiting 07:44:36 (5540): No heartbeat from core client for 30 sec - exiting 07:44:37 (5540): No heartbeat from core client for 30 sec - exiting 07:44:38 (5540): No heartbeat from core client for 30 sec - exiting 07:44:39 (5540): No heartbeat from core client for 30 sec - exiting 07:44:40 (5540): No heartbeat from core client for 30 sec - exiting 07:44:41 (5540): No heartbeat from core client for 30 sec - exiting 07:44:42 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:17:20 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:52:25 (4528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:07:21 (6340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:07:23 (6340): No heartbeat from core client for 30 sec - exiting 16:07:24 (6340): No heartbeat from core client for 30 sec - exiting 16:08:03 (904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:28:50 (6504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:28:51 (6504): No heartbeat from core client for 30 sec - exiting 17:28:52 (6504): No heartbeat from core client for 30 sec - exiting 17:28:53 (6504): No heartbeat from core client for 30 sec - exiting 17:30:24 (7216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:25 (7216): No heartbeat from core client for 30 sec - exiting 17:44:09 (6668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:44:50 (7820): No heartbeat from core client for 30 sec - exiting 17:44:51 (7820): No heartbeat from core client for 30 sec - exiting 17:44:53 (7820): No heartbeat from core client for 30 sec - exiting 17:44:54 (7820): No heartbeat from core client for 30 sec - exiting 17:44:55 (7820): No heartbeat from core client for 30 sec - exiting 17:44:56 (7820): No heartbeat from core client for 30 sec - exiting 17:44:57 (7820): No heartbeat from core client for 30 sec - exiting 17:44:58 (7820): No heartbeat from core client for 30 sec - exiting 17:44:59 (7820): No heartbeat from core client for 30 sec - exiting 17:45:00 (7820): No heartbeat from core client for 30 sec - exiting 17:45:01 (7820): No heartbeat from core client for 30 sec - exiting 17:45:02 (7820): No heartbeat from core client for 30 sec - exiting 17:45:03 (7820): No heartbeat from core client for 30 sec - exiting 17:45:05 (7820): No heartbeat from core client for 30 sec - exiting 17:45:06 (7820): No heartbeat from core client for 30 sec - exiting 17:45:07 (7820): No heartbeat from core client for 30 sec - exiting 17:45:08 (7820): No heartbeat from core client for 30 sec - exiting 17:45:09 (7820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:53 (7564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:59:59 (8276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:00:00 (8276): No heartbeat from core client for 30 sec - exiting 18:37:10 (8828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:37:46 (1100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:37:47 (1100): No heartbeat from core client for 30 sec - exiting 18:37:48 (1100): No heartbeat from core client for 30 sec - exiting 11:14:53 (5464): No heartbeat from core client for 30 sec - exiting 11:14:54 (5464): No heartbeat from core client for 30 sec - exiting 11:14:55 (5464): No heartbeat from core client for 30 sec - exiting 11:14:56 (5464): No heartbeat from core client for 30 sec - exiting 11:14:57 (5464): No heartbeat from core client for 30 sec - exiting 11:14:58 (5464): No heartbeat from core client for 30 sec - exiting 11:14:59 (5464): No heartbeat from core client for 30 sec - exiting 11:15:01 (5464): No heartbeat from core client for 30 sec - exiting 11:15:02 (5464): No heartbeat from core client for 30 sec - exiting 11:15:03 (5464): No heartbeat from core client for 30 sec - exiting 11:15:04 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:06:49 (8784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:43:48 (3456): No heartbeat from core client for 30 sec - exiting 07:43:49 (3456): No heartbeat from core client for 30 sec - exiting 07:43:51 (3456): No heartbeat from core client for 30 sec - exiting 07:43:52 (3456): No heartbeat from core client for 30 sec - exiting 07:43:53 (3456): No heartbeat from core client for 30 sec - exiting 07:43:54 (3456): No heartbeat from core client for 30 sec - exiting 07:43:55 (3456): No heartbeat from core client for 30 sec - exiting 07:43:56 (3456): No heartbeat from core client for 30 sec - exiting 07:43:57 (3456): No heartbeat from core client for 30 sec - exiting 07:43:58 (3456): No heartbeat from core client for 30 sec - exiting 07:43:59 (3456): No heartbeat from core client for 30 sec - exiting 07:44:00 (3456): No heartbeat from core client for 30 sec - exiting 07:44:01 (3456): No heartbeat from core client for 30 sec - exiting 07:44:03 (3456): No heartbeat from core client for 30 sec - exiting 07:44:04 (3456): No heartbeat from core client for 30 sec - exiting 07:44:05 (3456): No heartbeat from core client for 30 sec - exiting 07:44:06 (3456): No heartbeat from core client for 30 sec - exiting 07:44:07 (3456): No heartbeat from core client for 30 sec - exiting 07:44:08 (3456): No heartbeat from core client for 30 sec - exiting 07:44:09 (3456): No heartbeat from core client for 30 sec - exiting 07:44:10 (3456): No heartbeat from core client for 30 sec - exiting 07:44:11 (3456): No heartbeat from core client for 30 sec - exiting 07:44:12 (3456): No heartbeat from core client for 30 sec - exiting 07:44:13 (3456): No heartbeat from core client for 30 sec - exiting 07:44:14 (3456): No heartbeat from core client for 30 sec - exiting 07:44:15 (3456): No heartbeat from core client for 30 sec - exiting 07:44:16 (3456): No heartbeat from core client for 30 sec - exiting 07:44:17 (3456): No heartbeat from core client for 30 sec - exiting 07:44:18 (3456): No heartbeat from core client for 30 sec - exiting 07:44:19 (3456): No heartbeat from core client for 30 sec - exiting 07:44:20 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:38:30 (6360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:00:31 (5068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:16:59 (8284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:00:51 (2456): No heartbeat from core client for 30 sec - exiting 21:00:52 (2456): No heartbeat from core client for 30 sec - exiting 21:00:53 (2456): No heartbeat from core client for 30 sec - exiting 21:00:54 (2456): No heartbeat from core client for 30 sec - exiting 21:00:55 (2456): No heartbeat from core client for 30 sec - exiting 21:00:56 (2456): No heartbeat from core client for 30 sec - exiting 21:00:57 (2456): No heartbeat from core client for 30 sec - exiting 21:00:58 (2456): No heartbeat from core client for 30 sec - exiting 21:00:59 (2456): No heartbeat from core client for 30 sec - exiting 21:01:00 (2456): No heartbeat from core client for 30 sec - exiting 21:01:01 (2456): No heartbeat from core client for 30 sec - exiting 21:01:02 (2456): No heartbeat from core client for 30 sec - exiting 21:01:03 (2456): No heartbeat from core client for 30 sec - exiting 21:01:04 (2456): No heartbeat from core client for 30 sec - exiting 21:01:05 (2456): No heartbeat from core client for 30 sec - exiting 21:01:06 (2456): No heartbeat from core client for 30 sec - exiting 21:01:07 (2456): No heartbeat from core client for 30 sec - exiting 21:01:08 (2456): No heartbeat from core client for 30 sec - exiting 21:01:09 (2456): No heartbeat from core client for 30 sec - exiting 21:01:10 (2456): No heartbeat from core client for 30 sec - exiting 21:01:11 (2456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:12 (2456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 17:57:36 (9500): No heartbeat from core client for 30 sec - exiting 17:57:38 (9500): No heartbeat from core client for 30 sec - exiting 17:57:39 (9500): No heartbeat from core client for 30 sec - exiting 17:57:40 (9500): No heartbeat from core client for 30 sec - exiting 17:57:41 (9500): No heartbeat from core client for 30 sec - exiting 17:57:42 (9500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:57:43 (9500): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6644, iMonCtr=1 Model crash detected, will try to restart... 14:30:10 (4116): No heartbeat from core client for 30 sec - exiting 14:30:11 (4116): No heartbeat from core client for 30 sec - exiting 14:30:12 (4116): No heartbeat from core client for 30 sec - exiting 14:30:13 (4116): No heartbeat from core client for 30 sec - exiting 14:30:14 (4116): No heartbeat from core client for 30 sec - exiting 14:30:15 (4116): No heartbeat from core client for 30 sec - exiting 14:30:16 (4116): No heartbeat from core client for 30 sec - exiting 14:30:17 (4116): No heartbeat from core client for 30 sec - exiting 14:30:18 (4116): No heartbeat from core client for 30 sec - exiting 14:30:19 (4116): No heartbeat from core client for 30 sec - exiting 14:30:20 (4116): No heartbeat from core client for 30 sec - exiting 14:30:21 (4116): No heartbeat from core client for 30 sec - exiting 14:30:22 (4116): No heartbeat from core client for 30 sec - exiting 14:30:23 (4116): No heartbeat from core client for 30 sec - exiting 14:30:24 (4116): No heartbeat from core client for 30 sec - exiting 14:30:25 (4116): No heartbeat from core client for 30 sec - exiting 14:30:26 (4116): No heartbeat from core client for 30 sec - exiting 14:30:27 (4116): No heartbeat from core client for 30 sec - exiting 14:30:28 (4116): No heartbeat from core client for 30 sec - exiting 14:30:29 (4116): No heartbeat from core client for 30 sec - exiting 14:30:30 (4116): No heartbeat from core client for 30 sec - exiting 14:30:31 (4116): No heartbeat from core client for 30 sec - exiting 14:30:32 (4116): No heartbeat from core client for 30 sec - exiting 14:30:33 (4116): No heartbeat from core client for 30 sec - exiting 14:30:34 (4116): No heartbeat from core client for 30 sec - exiting 14:30:35 (4116): No heartbeat from core client for 30 sec - exiting 14:30:36 (4116): No heartbeat from core client for 30 sec - exiting 14:30:37 (4116): No heartbeat from core client for 30 sec - exiting 14:30:39 (4116): No heartbeat from core client for 30 sec - exiting 14:30:40 (4116): No heartbeat from core client for 30 sec - exiting 14:30:41 (4116): No heartbeat from core client for 30 sec - exiting 14:30:42 (4116): No heartbeat from core client for 30 sec - exiting 14:30:43 (4116): No heartbeat from core client for 30 sec - exiting 14:30:44 (4116): No heartbeat from core client for 30 sec - exiting 14:30:45 (4116): No heartbeat from core client for 30 sec - exiting 14:30:46 (4116): No heartbeat from core client for 30 sec - exiting 14:30:47 (4116): No heartbeat from core client for 30 sec - exiting 14:30:48 (4116): No heartbeat from core client for 30 sec - exiting 14:30:49 (4116): No heartbeat from core client for 30 sec - exiting 14:30:51 (4116): No heartbeat from core client for 30 sec - exiting 14:30:52 (4116): No heartbeat from core client for 30 sec - exiting 14:30:53 (4116): No heartbeat from core client for 30 sec - exiting 14:30:54 (4116): No heartbeat from core client for 30 sec - exiting 14:30:55 (4116): No heartbeat from core client for 30 sec - exiting 14:30:56 (4116): No heartbeat from core client for 30 sec - exiting 14:30:57 (4116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:16:54 (1064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:23:40 (6284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:41 (6284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 20:51:29 (7576): No heartbeat from core client for 30 sec - exiting 20:51:30 (7576): No heartbeat from core client for 30 sec - exiting 20:51:31 (7576): No heartbeat from core client for 30 sec - exiting 20:51:32 (7576): No heartbeat from core client for 30 sec - exiting 20:51:33 (7576): No heartbeat from core client for 30 sec - exiting 20:51:34 (7576): No heartbeat from core client for 30 sec - exiting 20:51:35 (7576): No heartbeat from core client for 30 sec - exiting 20:51:37 (7576): No heartbeat from core client for 30 sec - exiting 20:51:38 (7576): No heartbeat from core client for 30 sec - exiting 20:51:39 (7576): No heartbeat from core client for 30 sec - exiting 20:51:40 (7576): No heartbeat from core client for 30 sec - exiting 20:51:41 (7576): No heartbeat from core client for 30 sec - exiting 20:51:42 (7576): No heartbeat from core client for 30 sec - exiting 20:51:43 (7576): No heartbeat from core client for 30 sec - exiting 20:51:44 (7576): No heartbeat from core client for 30 sec - exiting 20:51:45 (7576): No heartbeat from core client for 30 sec - exiting 20:51:46 (7576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:51:48 (7576): No heartbeat from core client for 30 sec - exiting 13:54:04 (6556): No heartbeat from core client for 30 sec - exiting 13:54:05 (6556): No heartbeat from core client for 30 sec - exiting 13:54:06 (6556): No heartbeat from core client for 30 sec - exiting 13:54:07 (6556): No heartbeat from core client for 30 sec - exiting 13:54:08 (6556): No heartbeat from core client for 30 sec - exiting 13:54:09 (6556): No heartbeat from core client for 30 sec - exiting 13:54:10 (6556): No heartbeat from core client for 30 sec - exiting 13:54:11 (6556): No heartbeat from core client for 30 sec - exiting 13:54:12 (6556): No heartbeat from core client for 30 sec - exiting 13:54:14 (6556): No heartbeat from core client for 30 sec - exiting 13:54:15 (6556): No heartbeat from core client for 30 sec - exiting 13:54:16 (6556): No heartbeat from core client for 30 sec - exiting 13:54:17 (6556): No heartbeat from core client for 30 sec - exiting 13:54:18 (6556): No heartbeat from core client for 30 sec - exiting 13:54:19 (6556): No heartbeat from core client for 30 sec - exiting 13:54:20 (6556): No heartbeat from core client for 30 sec - exiting 13:54:21 (6556): No heartbeat from core client for 30 sec - exiting 13:54:22 (6556): No heartbeat from core client for 30 sec - exiting 13:54:23 (6556): No heartbeat from core client for 30 sec - exiting 13:54:24 (6556): No heartbeat from core client for 30 sec - exiting 13:54:26 (6556): No heartbeat from core client for 30 sec - exiting 13:54:27 (6556): No heartbeat from core client for 30 sec - exiting 13:54:28 (6556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:13:07 (1924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:13:08 (1924): No heartbeat from core client for 30 sec - exiting 15:13:09 (1924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:03:18 (9924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:12:46 (10320): Can't acquire lockfile (32) - waiting 35s Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 17:27:53	1036099	12918767	hadcm3n_o048_1940_40_007264533_0	259,200	470,577	1.8155
25 Jul 2011 12:57:04	1036099	12918767	hadcm3n_o048_1940_40_007264533_0	233,280	418,846	1.7955
25 Jul 2011 12:57:04	1036099	12918767	hadcm3n_o048_1940_40_007264533_0	207,360	366,675	1.7683
08 Jul 2011 19:59:12	1036099	12918767	hadcm3n_o048_1940_40_007264533_0	181,440	314,342	1.7325
04 Jul 2011 19:00:59	1036099	12918767	hadcm3n_o048_1940_40_007264533_0	155,520	261,856	1.6837
26 Jun 2011 14:16:27	1036099	12918767	hadcm3n_o048_1940_40_007264533_0	129,600	210,394	1.6234
10 Jun 2011 05:39:18	1036099	12918767	hadcm3n_o048_1940_40_007264533_0	103,680	165,802	1.5992
09 Jun 2011 18:24:58	1036099	12918767	hadcm3n_o048_1940_40_007264533_0	77,760	124,366	1.5994
08 Jun 2011 14:55:13	1036099	12918767	hadcm3n_o048_1940_40_007264533_0	51,840	83,821	1.6169
07 Jun 2011 18:17:47	1036099	12918767	hadcm3n_o048_1940_40_007264533_0	25,920	44,589	1.7203