Task 15440153

Name	hadcm3n_zkv1_1880_40_008245459_1
Workunit	8400583
Created	20 Nov 2012, 18:56:20 UTC
Sent	20 Nov 2012, 18:56:29 UTC
Report deadline	20 Feb 2013, 2:23:40 UTC
Received	11 Jan 2013, 17:48:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1036046
Run time	10 days 13 hours 16 min 25 sec
CPU time	8 days 14 hours 8 min 30 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	2.82 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> 01:46:23 (7664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:42:44 (27488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:39:07 (3588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:22:30 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:36:23 (2428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:11:17 (1656): No heartbeat from core client for 30 sec - exiting 20:11:18 (1656): No heartbeat from core client for 30 sec - exiting 20:11:19 (1656): No heartbeat from core client for 30 sec - exiting 20:11:20 (1656): No heartbeat from core client for 30 sec - exiting 20:11:21 (1656): No heartbeat from core client for 30 sec - exiting 20:11:22 (1656): No heartbeat from core client for 30 sec - exiting 20:11:23 (1656): No heartbeat from core client for 30 sec - exiting 20:11:24 (1656): No heartbeat from core client for 30 sec - exiting 20:11:25 (1656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:08:25 (14016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:38:52 (23220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:01:55 (5080): No heartbeat from core client for 30 sec - exiting 21:01:56 (5080): No heartbeat from core client for 30 sec - exiting 21:01:57 (5080): No heartbeat from core client for 30 sec - exiting 21:01:59 (5080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:09:32 (4556): No heartbeat from core client for 30 sec - exiting 18:09:33 (4556): No heartbeat from core client for 30 sec - exiting 18:09:34 (4556): No heartbeat from core client for 30 sec - exiting 18:09:35 (4556): No heartbeat from core client for 30 sec - exiting 18:09:36 (4556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:27:19 (1364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:29:36 (4628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:02:47 (508): No heartbeat from core client for 30 sec - exiting 20:02:48 (508): No heartbeat from core client for 30 sec - exiting 20:02:49 (508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:47:03 (4004): No heartbeat from core client for 30 sec - exiting 21:47:04 (4004): No heartbeat from core client for 30 sec - exiting 21:47:05 (4004): No heartbeat from core client for 30 sec - exiting 21:47:06 (4004): No heartbeat from core client for 30 sec - exiting 21:47:07 (4004): No heartbeat from core client for 30 sec - exiting 21:47:08 (4004): No heartbeat from core client for 30 sec - exiting 21:47:09 (4004): No heartbeat from core client for 30 sec - exiting 21:47:10 (4004): No heartbeat from core client for 30 sec - exiting 21:47:11 (4004): No heartbeat from core client for 30 sec - exiting 21:47:12 (4004): No heartbeat from core client for 30 sec - exiting 21:47:13 (4004): No heartbeat from core client for 30 sec - exiting 21:47:14 (4004): No heartbeat from core client for 30 sec - exiting 21:47:15 (4004): No heartbeat from core client for 30 sec - exiting 21:47:16 (4004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:37:59 (2312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:11:41 (380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4696, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3884, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 07:14:31 (3368): No heartbeat from core client for 30 sec - exiting 07:14:32 (3368): No heartbeat from core client for 30 sec - exiting 07:14:33 (3368): No heartbeat from core client for 30 sec - exiting 07:14:34 (3368): No heartbeat from core client for 30 sec - exiting 07:14:36 (3368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:25:23 (3892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:52:16 (3976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:44:17 (4220): No heartbeat from core client for 30 sec - exiting 08:44:18 (4220): No heartbeat from core client for 30 sec - exiting 08:44:19 (4220): No heartbeat from core client for 30 sec - exiting 08:44:20 (4220): No heartbeat from core client for 30 sec - exiting 08:44:21 (4220): No heartbeat from core client for 30 sec - exiting 08:44:22 (4220): No heartbeat from core client for 30 sec - exiting 08:44:23 (4220): No heartbeat from core client for 30 sec - exiting 08:44:24 (4220): No heartbeat from core client for 30 sec - exiting 08:44:25 (4220): No heartbeat from core client for 30 sec - exiting 08:44:26 (4220): No heartbeat from core client for 30 sec - exiting 08:44:27 (4220): No heartbeat from core client for 30 sec - exiting 08:44:28 (4220): No heartbeat from core client for 30 sec - exiting 08:44:29 (4220): No heartbeat from core client for 30 sec - exiting 08:44:30 (4220): No heartbeat from core client for 30 sec - exiting 08:44:31 (4220): No heartbeat from core client for 30 sec - exiting 08:44:32 (4220): No heartbeat from core client for 30 sec - exiting 08:44:33 (4220): No heartbeat from core client for 30 sec - exiting 08:44:34 (4220): No heartbeat from core client for 30 sec - exiting 08:44:35 (4220): No heartbeat from core client for 30 sec - exiting 08:44:36 (4220): No heartbeat from core client for 30 sec - exiting 08:44:37 (4220): No heartbeat from core client for 30 sec - exiting 08:44:38 (4220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:38:49 (4008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:03:20 (3660): No heartbeat from core client for 30 sec - exiting 19:03:21 (3660): No heartbeat from core client for 30 sec - exiting 19:03:22 (3660): No heartbeat from core client for 30 sec - exiting 19:03:23 (3660): No heartbeat from core client for 30 sec - exiting 19:03:24 (3660): No heartbeat from core client for 30 sec - exiting 19:03:25 (3660): No heartbeat from core client for 30 sec - exiting 19:03:26 (3660): No heartbeat from core client for 30 sec - exiting 19:03:27 (3660): No heartbeat from core client for 30 sec - exiting 19:03:28 (3660): No heartbeat from core client for 30 sec - exiting 19:03:29 (3660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:28:50 (4848): No heartbeat from core client for 30 sec - exiting 14:28:51 (4848): No heartbeat from core client for 30 sec - exiting 14:28:52 (4848): No heartbeat from core client for 30 sec - exiting 14:28:53 (4848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:18:16 (4864): No heartbeat from core client for 30 sec - exiting 12:18:18 (4864): No heartbeat from core client for 30 sec - exiting 12:18:19 (4864): No heartbeat from core client for 30 sec - exiting 12:18:20 (4864): No heartbeat from core client for 30 sec - exiting 12:18:21 (4864): No heartbeat from core client for 30 sec - exiting 12:18:22 (4864): No heartbeat from core client for 30 sec - exiting 12:18:23 (4864): No heartbeat from core client for 30 sec - exiting 12:18:24 (4864): No heartbeat from core client for 30 sec - exiting 12:18:25 (4864): No heartbeat from core client for 30 sec - exiting 12:18:26 (4864): No heartbeat from core client for 30 sec - exiting 12:18:27 (4864): No heartbeat from core client for 30 sec - exiting 12:18:28 (4864): No heartbeat from core client for 30 sec - exiting 12:18:29 (4864): No heartbeat from core client for 30 sec - exiting 12:18:30 (4864): No heartbeat from core client for 30 sec - exiting 12:18:31 (4864): No heartbeat from core client for 30 sec - exiting 12:18:32 (4864): No heartbeat from core client for 30 sec - exiting 12:18:33 (4864): No heartbeat from core client for 30 sec - exiting 12:18:34 (4864): No heartbeat from core client for 30 sec - exiting 12:18:35 (4864): No heartbeat from core client for 30 sec - exiting 12:18:36 (4864): No heartbeat from core client for 30 sec - exiting 12:18:37 (4864): No heartbeat from core client for 30 sec - exiting 12:18:38 (4864): No heartbeat from core client for 30 sec - exiting 12:18:39 (4864): No heartbeat from core client for 30 sec - exiting 12:18:40 (4864): No heartbeat from core client for 30 sec - exiting 12:18:41 (4864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:56:12 (4208): No heartbeat from core client for 30 sec - exiting 19:56:13 (4208): No heartbeat from core client for 30 sec - exiting 19:56:14 (4208): No heartbeat from core client for 30 sec - exiting 19:56:15 (4208): No heartbeat from core client for 30 sec - exiting 19:56:16 (4208): No heartbeat from core client for 30 sec - exiting 19:56:17 (4208): No heartbeat from core client for 30 sec - exiting 19:56:18 (4208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:46 (4136): No heartbeat from core client for 30 sec - exiting 18:56:47 (4136): No heartbeat from core client for 30 sec - exiting 18:56:48 (4136): No heartbeat from core client for 30 sec - exiting 18:56:49 (4136): No heartbeat from core client for 30 sec - exiting 18:56:50 (4136): No heartbeat from core client for 30 sec - exiting 18:56:51 (4136): No heartbeat from core client for 30 sec - exiting 18:56:52 (4136): No heartbeat from core client for 30 sec - exiting 18:56:53 (4136): No heartbeat from core client for 30 sec - exiting 18:56:54 (4136): No heartbeat from core client for 30 sec - exiting 18:56:55 (4136): No heartbeat from core client for 30 sec - exiting 18:56:56 (4136): No heartbeat from core client for 30 sec - exiting 18:56:57 (4136): No heartbeat from core client for 30 sec - exiting 18:56:58 (4136): No heartbeat from core client for 30 sec - exiting 18:56:59 (4136): No heartbeat from core client for 30 sec - exiting 18:57:00 (4136): No heartbeat from core client for 30 sec - exiting 18:57:01 (4136): No heartbeat from core client for 30 sec - exiting 18:57:02 (4136): No heartbeat from core client for 30 sec - exiting 18:57:03 (4136): No heartbeat from core client for 30 sec - exiting 18:57:04 (4136): No heartbeat from core client for 30 sec - exiting 18:57:05 (4136): No heartbeat from core client for 30 sec - exiting 18:57:06 (4136): No heartbeat from core client for 30 sec - exiting 18:57:07 (4136): No heartbeat from core client for 30 sec - exiting 18:57:08 (4136): No heartbeat from core client for 30 sec - exiting 18:57:09 (4136): No heartbeat from core client for 30 sec - exiting 18:57:10 (4136): No heartbeat from core client for 30 sec - exiting 18:57:11 (4136): No heartbeat from core client for 30 sec - exiting 18:57:12 (4136): No heartbeat from core client for 30 sec - exiting 18:57:13 (4136): No heartbeat from core client for 30 sec - exiting 18:57:14 (4136): No heartbeat from core client for 30 sec - exiting 18:57:15 (4136): No heartbeat from core client for 30 sec - exiting 18:57:17 (4136): No heartbeat from core client for 30 sec - exiting 18:57:18 (4136): No heartbeat from core client for 30 sec - exiting 18:57:19 (4136): No heartbeat from core client for 30 sec - exiting 18:57:20 (4136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:20:44 (4048): No heartbeat from core client for 30 sec - exiting 15:20:45 (4048): No heartbeat from core client for 30 sec - exiting 15:20:46 (4048): No heartbeat from core client for 30 sec - exiting 15:20:47 (4048): No heartbeat from core client for 30 sec - exiting 15:20:48 (4048): No heartbeat from core client for 30 sec - exiting 15:20:49 (4048): No heartbeat from core client for 30 sec - exiting 15:20:50 (4048): No heartbeat from core client for 30 sec - exiting 15:20:51 (4048): No heartbeat from core client for 30 sec - exiting 15:20:52 (4048): No heartbeat from core client for 30 sec - exiting 15:20:53 (4048): No heartbeat from core client for 30 sec - exiting 15:20:54 (4048): No heartbeat from core client for 30 sec - exiting 15:20:56 (4048): No heartbeat from core client for 30 sec - exiting 15:20:57 (4048): No heartbeat from core client for 30 sec - exiting 15:20:58 (4048): No heartbeat from core client for 30 sec - exiting 15:20:59 (4048): No heartbeat from core client for 30 sec - exiting 15:21:00 (4048): No heartbeat from core client for 30 sec - exiting 15:21:01 (4048): No heartbeat from core client for 30 sec - exiting 15:21:02 (4048): No heartbeat from core client for 30 sec - exiting 15:21:03 (4048): No heartbeat from core client for 30 sec - exiting 15:21:04 (4048): No heartbeat from core client for 30 sec - exiting 15:21:05 (4048): No heartbeat from core client for 30 sec - exiting 15:21:07 (4048): No heartbeat from core client for 30 sec - exiting 15:21:08 (4048): No heartbeat from core client for 30 sec - exiting 15:21:09 (4048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:34:26 (964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Jan 2013 02:08:11	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	596,160	719,498	1.2069
06 Jan 2013 19:15:37	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	570,240	686,689	1.2042
06 Jan 2013 06:30:40	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	544,320	653,642	1.2008
04 Jan 2013 15:35:45	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	518,400	620,937	1.1978
30 Dec 2012 23:46:28	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	492,480	588,124	1.1942
22 Dec 2012 20:19:04	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	466,560	555,139	1.1899
20 Dec 2012 19:11:31	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	440,640	522,205	1.1851
18 Dec 2012 18:16:25	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	414,720	489,309	1.1799
14 Dec 2012 09:40:55	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	388,800	471,461	1.2126
14 Dec 2012 00:03:25	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	362,880	460,627	1.2694
14 Dec 2012 00:03:25	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	336,960	427,898	1.2699
07 Dec 2012 22:54:17	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	311,040	394,559	1.2685
03 Dec 2012 23:34:49	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	285,120	361,722	1.2687
03 Dec 2012 06:03:50	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	259,200	328,623	1.2678
01 Dec 2012 22:56:36	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	233,280	295,766	1.2679
30 Nov 2012 06:06:12	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	207,360	263,082	1.2687
29 Nov 2012 07:07:08	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	181,440	230,418	1.2699
28 Nov 2012 03:55:42	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	155,520	197,582	1.2705
26 Nov 2012 23:41:44	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	129,600	164,860	1.2721
25 Nov 2012 08:15:19	1036046	15440153	hadcm3n_zkv1_1880_40_008245459_1	103,680	132,115	1.2743