Task 16289752

Name	hadcm3n_7yqz_1980_40_008456318_3
Workunit	8607174
Created	11 Feb 2014, 0:25:25 UTC
Sent	11 Feb 2014, 0:25:31 UTC
Report deadline	13 May 2014, 7:52:42 UTC
Received	2 Mar 2014, 1:11:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1296476
Run time	6 days 20 hours 45 min 11 sec
CPU time	6 days 16 hours 38 min 15 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	3.78 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 12:31:21 (7648): Can't set up shared mem: -1. Will run in standalone mode. 12:31:21 (6728): Can't set up shared mem: -1. Will run in standalone mode. 12:32:54 (9468): Can't acquire lockfile (32) - waiting 35s 12:33:29 (9468): Can't acquire lockfile (32) - exiting 12:33:29 (9468): Error: The process cannot access the file because it is being used by another process. (0x20) 12:43:52 (1620): Can't acquire lockfile (32) - waiting 35s 12:44:27 (1620): Can't acquire lockfile (32) - exiting 12:44:27 (1620): Error: The process cannot access the file because it is being used by another process. (0x20) 12:54:43 (1404): Can't acquire lockfile (32) - waiting 35s 12:55:18 (1404): Can't acquire lockfile (32) - exiting 12:55:18 (1404): Error: The process cannot access the file because it is being used by another process. (0x20) 13:06:33 (676): Can't acquire lockfile (32) - waiting 35s 13:07:08 (676): Can't acquire lockfile (32) - exiting 13:07:08 (676): Error: The process cannot access the file because it is being used by another process. (0x20) 13:17:15 (2112): Can't acquire lockfile (32) - waiting 35s 13:17:50 (2112): Can't acquire lockfile (32) - exiting 13:17:50 (2112): Error: The process cannot access the file because it is being used by another process. (0x20) 13:29:30 (5544): Can't acquire lockfile (32) - waiting 35s 13:30:05 (5544): Can't acquire lockfile (32) - exiting 13:30:05 (5544): Error: The process cannot access the file because it is being used by another process. (0x20) 13:44:05 (8980): Can't acquire lockfile (32) - waiting 35s 13:44:40 (8980): Can't acquire lockfile (32) - exiting 13:44:40 (8980): Error: The process cannot access the file because it is being used by another process. (0x20) 13:57:51 (10904): Can't acquire lockfile (32) - waiting 35s 13:58:26 (10904): Can't acquire lockfile (32) - exiting 13:58:26 (10904): Error: The process cannot access the file because it is being used by another process. (0x20) 16:09:35 (8060): Can't acquire lockfile (32) - waiting 35s 16:10:10 (8060): Can't acquire lockfile (32) - exiting 16:10:10 (8060): Error: The process cannot access the file because it is being used by another process. (0x20) 16:20:34 (7396): Can't acquire lockfile (32) - waiting 35s 16:21:09 (7396): Can't acquire lockfile (32) - exiting 16:21:09 (7396): Error: The process cannot access the file because it is being used by another process. (0x20) 16:31:51 (11880): Can't acquire lockfile (32) - waiting 35s 16:32:26 (11880): Can't acquire lockfile (32) - exiting 16:32:26 (11880): Error: The process cannot access the file because it is being used by another process. (0x20) 16:42:41 (12080): Can't acquire lockfile (32) - waiting 35s 16:43:16 (12080): Can't acquire lockfile (32) - exiting 16:43:16 (12080): Error: The process cannot access the file because it is being used by another process. (0x20) 16:53:33 (11316): Can't acquire lockfile (32) - waiting 35s 16:54:08 (11316): Can't acquire lockfile (32) - exiting 16:54:08 (11316): Error: The process cannot access the file because it is being used by another process. (0x20) 17:07:42 (11860): Can't acquire lockfile (32) - waiting 35s 17:08:17 (11860): Can't acquire lockfile (32) - exiting 17:08:17 (11860): Error: The process cannot access the file because it is being used by another process. (0x20) 17:18:59 (11444): Can't acquire lockfile (32) - waiting 35s 17:19:34 (11444): Can't acquire lockfile (32) - exiting 17:19:34 (11444): Error: The process cannot access the file because it is being used by another process. (0x20) 17:29:48 (8812): Can't acquire lockfile (32) - waiting 35s 17:30:23 (8812): Can't acquire lockfile (32) - exiting 17:30:23 (8812): Error: The process cannot access the file because it is being used by another process. (0x20) 17:43:13 (12188): Can't acquire lockfile (32) - waiting 35s 17:43:48 (12188): Can't acquire lockfile (32) - exiting 17:43:48 (12188): Error: The process cannot access the file because it is being used by another process. (0x20) 17:54:27 (5512): Can't acquire lockfile (32) - waiting 35s 17:55:02 (5512): Can't acquire lockfile (32) - exiting 17:55:02 (5512): Error: The process cannot access the file because it is being used by another process. (0x20) 18:05:23 (6560): Can't acquire lockfile (32) - waiting 35s 18:05:58 (6560): Can't acquire lockfile (32) - exiting 18:05:58 (6560): Error: The process cannot access the file because it is being used by another process. (0x20) 18:20:44 (6576): Can't acquire lockfile (32) - waiting 35s 18:21:19 (6576): Can't acquire lockfile (32) - exiting 18:21:19 (6576): Error: The process cannot access the file because it is being used by another process. (0x20) 18:44:19 (12188): Can't acquire lockfile (32) - waiting 35s 18:44:54 (12188): Can't acquire lockfile (32) - exiting 18:44:54 (12188): Error: The process cannot access the file because it is being used by another process. (0x20) 19:02:56 (7312): Can't acquire lockfile (32) - waiting 35s 19:03:31 (7312): Can't acquire lockfile (32) - exiting 19:03:31 (7312): Error: The process cannot access the file because it is being used by another process. (0x20) 19:18:09 (11368): Can't acquire lockfile (32) - waiting 35s 19:18:44 (11368): Can't acquire lockfile (32) - exiting 19:18:44 (11368): Error: The process cannot access the file because it is being used by another process. (0x20) 19:29:12 (11020): Can't acquire lockfile (32) - waiting 35s 19:29:47 (11020): Can't acquire lockfile (32) - exiting 19:29:47 (11020): Error: The process cannot access the file because it is being used by another process. (0x20) 19:45:40 (9528): Can't acquire lockfile (32) - waiting 35s 19:46:15 (9528): Can't acquire lockfile (32) - exiting 19:46:15 (9528): Error: The process cannot access the file because it is being used by another process. (0x20) 20:18:49 (2320): Can't acquire lockfile (32) - waiting 35s 20:19:24 (2320): Can't acquire lockfile (32) - exiting 20:19:24 (2320): Error: The process cannot access the file because it is being used by another process. (0x20) 20:30:36 (11008): Can't acquire lockfile (32) - waiting 35s 20:31:11 (11008): Can't acquire lockfile (32) - exiting 20:31:11 (11008): Error: The process cannot access the file because it is being used by another process. (0x20) 20:46:50 (1244): Can't acquire lockfile (32) - waiting 35s 20:47:25 (1244): Can't acquire lockfile (32) - exiting 20:47:25 (1244): Error: The process cannot access the file because it is being used by another process. (0x20) 21:03:02 (1596): Can't acquire lockfile (32) - waiting 35s 21:03:37 (1596): Can't acquire lockfile (32) - exiting 21:03:37 (1596): Error: The process cannot access the file because it is being used by another process. (0x20) 21:33:11 (12236): Can't acquire lockfile (32) - waiting 35s 21:33:46 (12236): Can't acquire lockfile (32) - exiting 21:33:46 (12236): Error: The process cannot access the file because it is being used by another process. (0x20) 21:43:46 (5512): Can't acquire lockfile (32) - waiting 35s 21:44:21 (5512): Can't acquire lockfile (32) - exiting 21:44:21 (5512): Error: The process cannot access the file because it is being used by another process. (0x20) 22:04:20 (12112): Can't acquire lockfile (32) - waiting 35s 22:04:55 (12112): Can't acquire lockfile (32) - exiting 22:04:55 (12112): Error: The process cannot access the file because it is being used by another process. (0x20) 22:33:48 (3292): Can't acquire lockfile (32) - waiting 35s 22:34:23 (3292): Can't acquire lockfile (32) - exiting 22:34:23 (3292): Error: The process cannot access the file because it is being used by another process. (0x20) 22:45:19 (6048): Can't acquire lockfile (32) - waiting 35s 22:45:54 (6048): Can't acquire lockfile (32) - exiting 22:45:54 (6048): Error: The process cannot access the file because it is being used by another process. (0x20) 22:56:26 (5504): Can't acquire lockfile (32) - waiting 35s 22:57:01 (5504): Can't acquire lockfile (32) - exiting 22:57:01 (5504): Error: The process cannot access the file because it is being used by another process. (0x20) 23:12:40 (4404): Can't acquire lockfile (32) - waiting 35s 23:13:15 (4404): Can't acquire lockfile (32) - exiting 23:13:15 (4404): Error: The process cannot access the file because it is being used by another process. (0x20) 23:25:40 (8368): Can't acquire lockfile (32) - waiting 35s 23:26:15 (8368): Can't acquire lockfile (32) - exiting 23:26:15 (8368): Error: The process cannot access the file because it is being used by another process. (0x20) 23:57:04 (8348): Can't acquire lockfile (32) - waiting 35s 23:57:39 (8348): Can't acquire lockfile (32) - exiting 23:57:39 (8348): Error: The process cannot access the file because it is being used by another process. (0x20) 00:08:23 (6108): Can't acquire lockfile (32) - waiting 35s 00:08:58 (6108): Can't acquire lockfile (32) - exiting 00:08:58 (6108): Error: The process cannot access the file because it is being used by another process. (0x20) 00:22:58 (11144): Can't acquire lockfile (32) - waiting 35s 00:23:33 (11144): Can't acquire lockfile (32) - exiting 00:23:33 (11144): Error: The process cannot access the file because it is being used by another process. (0x20) 01:25:45 (1168): Can't acquire lockfile (32) - waiting 35s 01:26:20 (1168): Can't acquire lockfile (32) - exiting 01:26:20 (1168): Error: The process cannot access the file because it is being used by another process. (0x20) 01:36:50 (11960): Can't acquire lockfile (32) - waiting 35s 01:37:25 (11960): Can't acquire lockfile (32) - exiting 01:37:25 (11960): Error: The process cannot access the file because it is being used by another process. (0x20) 01:50:57 (3196): Can't acquire lockfile (32) - waiting 35s 01:51:32 (3196): Can't acquire lockfile (32) - exiting 01:51:32 (3196): Error: The process cannot access the file because it is being used by another process. (0x20) 02:01:56 (4556): Can't acquire lockfile (32) - waiting 35s 02:02:31 (4556): Can't acquire lockfile (32) - exiting 02:02:31 (4556): Error: The process cannot access the file because it is being used by another process. (0x20) 12:23:03 (11600): Can't acquire lockfile (32) - waiting 35s 12:23:38 (11600): Can't acquire lockfile (32) - exiting 12:23:38 (11600): Error: The process cannot access the file because it is being used by another process. (0x20) CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:42:41 (1640): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:31:12 (6292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5672, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5672, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5672, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5672, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5672, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6696, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Feb 2014 10:44:56	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	1,010,880	570,407	0.5643
26 Feb 2014 06:28:55	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	984,960	554,594	0.5631
26 Feb 2014 02:03:16	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	959,040	539,302	0.5623
25 Feb 2014 21:47:39	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	933,120	524,246	0.5618
25 Feb 2014 17:16:22	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	907,200	507,972	0.5599
25 Feb 2014 05:03:51	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	881,280	492,806	0.5592
25 Feb 2014 00:28:18	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	855,360	477,587	0.5583
24 Feb 2014 11:39:17	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	829,440	462,445	0.5575
24 Feb 2014 07:18:47	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	803,520	447,000	0.5563
24 Feb 2014 02:53:13	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	777,600	431,595	0.5550
23 Feb 2014 21:22:31	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	751,680	416,577	0.5542
23 Feb 2014 07:10:40	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	725,760	401,363	0.5530
22 Feb 2014 11:22:49	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	699,840	386,325	0.5520
22 Feb 2014 00:31:25	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	673,920	370,962	0.5505
21 Feb 2014 01:41:29	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	648,000	355,066	0.5479
20 Feb 2014 06:31:23	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	622,080	339,648	0.5460
20 Feb 2014 02:10:43	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	596,160	324,252	0.5439
19 Feb 2014 10:07:51	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	570,240	308,834	0.5416
19 Feb 2014 02:51:56	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	544,320	293,476	0.5392
18 Feb 2014 12:03:47	1296476	16289752	hadcm3n_7yqz_1980_40_008456318_3	518,400	278,012	0.5363