Task 14655872

Name	hadcm3n_u0i0_1980_40_007833139_4
Workunit	7988251
Created	10 May 2012, 20:26:02 UTC
Sent	10 May 2012, 20:29:19 UTC
Report deadline	10 Aug 2012, 3:56:30 UTC
Received	4 Nov 2012, 4:23:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1178407
Run time	12 days 15 hours 57 min 49 sec
CPU time	9 days 3 hours 59 min 14 sec
Validate state	Invalid
Credit	11,197.44
Device peak FLOPS	3.37 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3980, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:16:19 (4840): No heartbeat from core client for 30 sec - exiting 09:16:20 (4840): No heartbeat from core client for 30 sec - exiting 09:16:21 (4840): No heartbeat from core client for 30 sec - exiting 09:16:22 (4840): No heartbeat from core client for 30 sec - exiting 09:16:23 (4840): No heartbeat from core client for 30 sec - exiting 09:16:24 (4840): No heartbeat from core client for 30 sec - exiting 09:16:25 (4840): No heartbeat from core client for 30 sec - exiting 09:16:26 (4840): No heartbeat from core client for 30 sec - exiting 09:16:28 (4840): No heartbeat from core client for 30 sec - exiting 09:16:29 (4840): No heartbeat from core client for 30 sec - exiting 09:16:30 (4840): No heartbeat from core client for 30 sec - exiting 09:16:31 (4840): No heartbeat from core client for 30 sec - exiting 09:16:32 (4840): No heartbeat from core client for 30 sec - exiting 09:16:33 (4840): No heartbeat from core client for 30 sec - exiting 09:16:34 (4840): No heartbeat from core client for 30 sec - exiting 09:16:35 (4840): No heartbeat from core client for 30 sec - exiting 09:16:36 (4840): No heartbeat from core client for 30 sec - exiting 09:16:37 (4840): No heartbeat from core client for 30 sec - exiting 09:16:39 (4840): No heartbeat from core client for 30 sec - exiting 09:16:40 (4840): No heartbeat from core client for 30 sec - exiting 09:16:41 (4840): No heartbeat from core client for 30 sec - exiting 09:16:42 (4840): No heartbeat from core client for 30 sec - exiting 09:16:43 (4840): No heartbeat from core client for 30 sec - exiting 09:16:44 (4840): No heartbeat from core client for 30 sec - exiting 09:16:45 (4840): No heartbeat from core client for 30 sec - exiting 09:16:46 (4840): No heartbeat from core client for 30 sec - exiting 09:16:47 (4840): No heartbeat from core client for 30 sec - exiting 09:16:49 (4840): No heartbeat from core client for 30 sec - exiting 09:16:50 (4840): No heartbeat from core client for 30 sec - exiting 09:16:51 (4840): No heartbeat from core client for 30 sec - exiting 09:16:52 (4840): No heartbeat from core client for 30 sec - exiting 09:16:53 (4840): No heartbeat from core client for 30 sec - exiting 09:16:54 (4840): No heartbeat from core client for 30 sec - exiting 09:16:55 (4840): No heartbeat from core client for 30 sec - exiting 09:16:56 (4840): No heartbeat from core client for 30 sec - exiting 09:16:57 (4840): No heartbeat from core client for 30 sec - exiting 09:16:58 (4840): No heartbeat from core client for 30 sec - exiting 09:16:59 (4840): No heartbeat from core client for 30 sec - exiting 09:17:00 (4840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:02:56 (2040): No heartbeat from core client for 30 sec - exiting 13:02:58 (2040): No heartbeat from core client for 30 sec - exiting 13:02:59 (2040): No heartbeat from core client for 30 sec - exiting 13:03:00 (2040): No heartbeat from core client for 30 sec - exiting 13:03:01 (2040): No heartbeat from core client for 30 sec - exiting 13:03:02 (2040): No heartbeat from core client for 30 sec - exiting 13:03:03 (2040): No heartbeat from core client for 30 sec - exiting 13:03:04 (2040): No heartbeat from core client for 30 sec - exiting 13:03:05 (2040): No heartbeat from core client for 30 sec - exiting 13:03:06 (2040): No heartbeat from core client for 30 sec - exiting 13:03:07 (2040): No heartbeat from core client for 30 sec - exiting 13:03:08 (2040): No heartbeat from core client for 30 sec - exiting 13:03:09 (2040): No heartbeat from core client for 30 sec - exiting 13:03:10 (2040): No heartbeat from core client for 30 sec - exiting 13:03:11 (2040): No heartbeat from core client for 30 sec - exiting 13:03:12 (2040): No heartbeat from core client for 30 sec - exiting 13:03:13 (2040): No heartbeat from core client for 30 sec - exiting 13:03:14 (2040): No heartbeat from core client for 30 sec - exiting 13:03:15 (2040): No heartbeat from core client for 30 sec - exiting 13:03:16 (2040): No heartbeat from core client for 30 sec - exiting 13:03:17 (2040): No heartbeat from core client for 30 sec - exiting 13:03:18 (2040): No heartbeat from core client for 30 sec - exiting 13:03:19 (2040): No heartbeat from core client for 30 sec - exiting 13:03:20 (2040): No heartbeat from core client for 30 sec - exiting 13:03:21 (2040): No heartbeat from core client for 30 sec - exiting 13:03:22 (2040): No heartbeat from core client for 30 sec - exiting 13:03:23 (2040): No heartbeat from core client for 30 sec - exiting 13:03:24 (2040): No heartbeat from core client for 30 sec - exiting 13:03:25 (2040): No heartbeat from core client for 30 sec - exiting 13:03:26 (2040): No heartbeat from core client for 30 sec - exiting 13:03:28 (2040): No heartbeat from core client for 30 sec - exiting 13:03:29 (2040): No heartbeat from core client for 30 sec - exiting 13:03:30 (2040): No heartbeat from core client for 30 sec - exiting 13:03:31 (2040): No heartbeat from core client for 30 sec - exiting 13:03:32 (2040): No heartbeat from core client for 30 sec - exiting 13:03:33 (2040): No heartbeat from core client for 30 sec - exiting 13:03:34 (2040): No heartbeat from core client for 30 sec - exiting 13:03:35 (2040): No heartbeat from core client for 30 sec - exiting 13:03:36 (2040): No heartbeat from core client for 30 sec - exiting 13:03:37 (2040): No heartbeat from core client for 30 sec - exiting 13:03:38 (2040): No heartbeat from core client for 30 sec - exiting 13:03:40 (2040): No heartbeat from core client for 30 sec - exiting 13:03:41 (2040): No heartbeat from core client for 30 sec - exiting 13:03:42 (2040): No heartbeat from core client for 30 sec - exiting 13:03:43 (2040): No heartbeat from core client for 30 sec - exiting 13:03:44 (2040): No heartbeat from core client for 30 sec - exiting 13:03:45 (2040): No heartbeat from core client for 30 sec - exiting 13:03:46 (2040): No heartbeat from core client for 30 sec - exiting 13:03:47 (2040): No heartbeat from core client for 30 sec - exiting 13:03:48 (2040): No heartbeat from core client for 30 sec - exiting 13:03:49 (2040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:38:57 (888): No heartbeat from core client for 30 sec - exiting 20:38:58 (888): No heartbeat from core client for 30 sec - exiting 20:38:59 (888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:47:29 (5616): No heartbeat from core client for 30 sec - exiting 23:47:30 (5616): No heartbeat from core client for 30 sec - exiting 23:47:31 (5616): No heartbeat from core client for 30 sec - exiting 23:47:32 (5616): No heartbeat from core client for 30 sec - exiting 23:47:33 (5616): No heartbeat from core client for 30 sec - exiting 23:47:34 (5616): No heartbeat from core client for 30 sec - exiting 23:47:35 (5616): No heartbeat from core client for 30 sec - exiting 23:47:36 (5616): No heartbeat from core client for 30 sec - exiting 23:47:37 (5616): No heartbeat from core client for 30 sec - exiting 23:47:38 (5616): No heartbeat from core client for 30 sec - exiting 23:47:39 (5616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:31:59 (1100): No heartbeat from core client for 30 sec - exiting 02:32:00 (1100): No heartbeat from core client for 30 sec - exiting 02:32:01 (1100): No heartbeat from core client for 30 sec - exiting 02:32:02 (1100): No heartbeat from core client for 30 sec - exiting 02:32:03 (1100): No heartbeat from core client for 30 sec - exiting 02:32:04 (1100): No heartbeat from core client for 30 sec - exiting 02:32:05 (1100): No heartbeat from core client for 30 sec - exiting 02:32:06 (1100): No heartbeat from core client for 30 sec - exiting 02:32:07 (1100): No heartbeat from core client for 30 sec - exiting 02:32:08 (1100): No heartbeat from core client for 30 sec - exiting 02:32:09 (1100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5036, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5036, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:41:19 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:48:06 (4788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Nov 2012 03:24:53	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	933,120	851,676	0.9127
03 Nov 2012 19:41:55	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	907,200	827,577	0.9122
03 Nov 2012 11:25:08	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	881,280	804,004	0.9123
21 Jul 2012 18:07:17	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	855,360	780,029	0.9119
21 Jul 2012 07:21:04	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	829,440	754,727	0.9099
12 Jun 2012 14:47:45	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	803,520	729,443	0.9078
11 Jun 2012 08:14:48	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	777,600	705,929	0.9078
10 Jun 2012 21:23:49	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	751,680	682,654	0.9082
10 Jun 2012 12:16:56	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	725,760	659,526	0.9087
05 Jun 2012 10:44:17	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	699,840	636,070	0.9089
04 Jun 2012 21:41:57	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	673,920	611,710	0.9077
04 Jun 2012 05:35:28	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	648,000	588,491	0.9082
03 Jun 2012 20:42:31	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	622,080	566,684	0.9110
03 Jun 2012 11:10:10	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	596,160	543,460	0.9116
03 Jun 2012 00:20:58	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	570,240	519,331	0.9107
02 Jun 2012 16:34:16	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	544,320	496,179	0.9116
01 Jun 2012 20:37:02	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	518,400	473,074	0.9126
01 Jun 2012 02:49:23	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	492,480	449,241	0.9122
30 May 2012 08:54:24	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	466,560	424,839	0.9106
30 May 2012 01:13:06	1178407	14655872	hadcm3n_u0i0_1980_40_007833139_4	440,640	400,971	0.9100