Task 13401128

Name	hadcm3n_o0qf_1940_40_007446866_2
Workunit	7644369
Created	19 Sep 2011, 15:59:16 UTC
Sent	19 Sep 2011, 16:06:40 UTC
Report deadline	19 Dec 2011, 23:33:51 UTC
Received	29 Oct 2011, 15:10:25 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1088808
Run time	10 days 14 hours 29 min 0 sec
CPU time	9 days 3 hours 45 min 2 sec
Validate state	Invalid
Credit	4,665.60
Device peak FLOPS	2.30 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:03:05 (7896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:04:04 (10096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:01:14 (9004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:05:19 (8512): No heartbeat from core client for 30 sec - exiting 00:05:20 (8512): No heartbeat from core client for 30 sec - exiting 00:05:21 (8512): No heartbeat from core client for 30 sec - exiting 00:05:22 (8512): No heartbeat from core client for 30 sec - exiting 00:05:23 (8512): No heartbeat from core client for 30 sec - exiting 00:05:24 (8512): No heartbeat from core client for 30 sec - exiting 00:05:25 (8512): No heartbeat from core client for 30 sec - exiting 00:05:26 (8512): No heartbeat from core client for 30 sec - exiting 00:05:27 (8512): No heartbeat from core client for 30 sec - exiting 00:05:28 (8512): No heartbeat from core client for 30 sec - exiting 00:05:29 (8512): No heartbeat from core client for 30 sec - exiting 00:05:30 (8512): No heartbeat from core client for 30 sec - exiting 00:05:31 (8512): No heartbeat from core client for 30 sec - exiting 00:05:32 (8512): No heartbeat from core client for 30 sec - exiting 00:05:33 (8512): No heartbeat from core client for 30 sec - exiting 00:05:34 (8512): No heartbeat from core client for 30 sec - exiting 00:05:35 (8512): No heartbeat from core client for 30 sec - exiting 00:05:36 (8512): No heartbeat from core client for 30 sec - exiting 00:05:37 (8512): No heartbeat from core client for 30 sec - exiting 00:05:38 (8512): No heartbeat from core client for 30 sec - exiting 00:05:39 (8512): No heartbeat from core client for 30 sec - exiting 00:05:40 (8512): No heartbeat from core client for 30 sec - exiting 00:05:41 (8512): No heartbeat from core client for 30 sec - exiting 00:05:42 (8512): No heartbeat from core client for 30 sec - exiting 00:05:43 (8512): No heartbeat from core client for 30 sec - exiting 00:05:44 (8512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:13:28 (5392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:13:30 (5392): No heartbeat from core client for 30 sec - exiting 00:15:02 (8888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:18:29 (4560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:18:32 (4560): No heartbeat from core client for 30 sec - exiting 00:18:33 (4560): No heartbeat from core client for 30 sec - exiting 00:19:37 (6916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:08:14 (12068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:53:11 (10116): No heartbeat from core client for 30 sec - exiting 10:53:12 (10116): No heartbeat from core client for 30 sec - exiting 10:53:13 (10116): No heartbeat from core client for 30 sec - exiting 10:53:14 (10116): No heartbeat from core client for 30 sec - exiting 10:53:15 (10116): No heartbeat from core client for 30 sec - exiting 10:53:16 (10116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:52:03 (8904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o0qfko.pje6c10 Error converting file to netcdf: dataout/o0qfko.pie6c10 Error converting file to netcdf: dataout/o0qfko.pfe6c10 Error converting file to netcdf: dataout/o0qfka.phe6c10 Error converting file to netcdf: dataout/o0qfka.pge6c10 Error converting file to netcdf: dataout/o0qfka.pee6c10 Error converting file to netcdf: dataout/o0qfka.pde6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:32:13 (7216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:29:13 (12088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:26:16 (15064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:26:54 (10300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o0qfko.pje9c10 Error converting file to netcdf: dataout/o0qfko.pie9c10 Error converting file to netcdf: dataout/o0qfko.pfe9c10 Error converting file to netcdf: dataout/o0qfka.phe9c10 Error converting file to netcdf: dataout/o0qfka.pge9c10 Error converting file to netcdf: dataout/o0qfka.pee9c10 Error converting file to netcdf: dataout/o0qfka.pde9c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:20:31 (14624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:21:16 (6288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:40:18 (10080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:40:52 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:17:43 (10680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:17:46 (10680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:42:35 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:50:43 (4180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6628, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6628, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6628, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6628, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish 07:48:42 (8124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:43:10 (15780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:11:00 (15780): No heartbeat from core client for 30 sec - exiting 12:13:51 (15780): No heartbeat from core client for 30 sec - exiting 12:16:59 (15780): No heartbeat from core client for 30 sec - exiting 12:20:06 (15780): No heartbeat from core client for 30 sec - exiting 12:22:57 (15780): No heartbeat from core client for 30 sec - exiting 12:25:49 (15780): No heartbeat from core client for 30 sec - exiting 12:28:41 (15780): No heartbeat from core client for 30 sec - exiting 12:31:32 (15780): No heartbeat from core client for 30 sec - exiting 12:34:24 (15780): No heartbeat from core client for 30 sec - exiting 12:37:16 (15780): No heartbeat from core client for 30 sec - exiting 12:40:07 (15780): No heartbeat from core client for 30 sec - exiting 12:42:59 (15780): No heartbeat from core client for 30 sec - exiting 12:45:50 (15780): No heartbeat from core client for 30 sec - exiting 18:48:59 (6972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14176, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14176, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish 09:48:31 (14176): No heartbeat from core client for 30 sec - exiting 09:48:32 (14176): No heartbeat from core client for 30 sec - exiting </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Oct 2011 13:06:21	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	388,800	757,926	1.9494
10 Oct 2011 19:07:09	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	362,880	706,888	1.9480
09 Oct 2011 18:03:54	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	336,960	656,730	1.9490
08 Oct 2011 18:53:32	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	311,040	606,367	1.9495
07 Oct 2011 17:45:14	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	285,120	555,860	1.9496
06 Oct 2011 08:48:18	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	259,200	505,173	1.9490
05 Oct 2011 08:29:49	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	233,280	455,217	1.9514
04 Oct 2011 00:28:56	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	207,360	405,874	1.9573
02 Oct 2011 08:55:49	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	181,440	354,742	1.9551
01 Oct 2011 00:25:16	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	155,520	305,267	1.9629
28 Sep 2011 20:58:08	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	129,600	253,546	1.9564
25 Sep 2011 05:53:55	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	103,680	202,511	1.9532
23 Sep 2011 15:56:10	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	77,760	151,851	1.9528
22 Sep 2011 06:31:16	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	51,840	100,352	1.9358
20 Sep 2011 17:50:34	1088808	13401128	hadcm3n_o0qf_1940_40_007446866_2	25,920	50,605	1.9524