Task 16053406

Name	hadcm3n_o2d1_2020_40_008376238_3
Workunit	8527097
Created	2 Oct 2013, 7:53:11 UTC
Sent	2 Oct 2013, 8:00:43 UTC
Report deadline	1 Jan 2014, 15:27:54 UTC
Received	26 Oct 2013, 12:15:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	900317
Run time	17 days 1 hours 46 min 35 sec
CPU time	17 days 1 hours 46 min 35 sec
Validate state	Invalid
Credit	5,598.72
Device peak FLOPS	1.34 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.2.17</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:16:20 (39936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:49:01 (40452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:49:48 (39228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:26:25 (61140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:21:06 (75180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:39:24 (82844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:32:55 (86764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:33:48 (91772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:04:47 (91944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:07:12 (108136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:52:45 (108256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:50:31 (110824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:27 (120764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:28 (120764): No heartbeat from core client for 30 sec - exiting 17:01:59 (123724): No heartbeat from core client for 30 sec - exiting 17:02:00 (123724): No heartbeat from core client for 30 sec - exiting 17:02:01 (123724): No heartbeat from core client for 30 sec - exiting 17:02:02 (123724): No heartbeat from core client for 30 sec - exiting 17:02:03 (123724): No heartbeat from core client for 30 sec - exiting 17:02:04 (123724): No heartbeat from core client for 30 sec - exiting 17:02:05 (123724): No heartbeat from core client for 30 sec - exiting 17:02:06 (123724): No heartbeat from core client for 30 sec - exiting 17:02:07 (123724): No heartbeat from core client for 30 sec - exiting 17:02:08 (123724): No heartbeat from core client for 30 sec - exiting 17:02:09 (123724): No heartbeat from core client for 30 sec - exiting 17:02:10 (123724): No heartbeat from core client for 30 sec - exiting 17:02:11 (123724): No heartbeat from core client for 30 sec - exiting 17:02:12 (123724): No heartbeat from core client for 30 sec - exiting 17:02:13 (123724): No heartbeat from core client for 30 sec - exiting 17:02:14 (123724): No heartbeat from core client for 30 sec - exiting 17:02:15 (123724): No heartbeat from core client for 30 sec - exiting 17:02:16 (123724): No heartbeat from core client for 30 sec - exiting 17:02:17 (123724): No heartbeat from core client for 30 sec - exiting 17:02:18 (123724): No heartbeat from core client for 30 sec - exiting 17:02:19 (123724): No heartbeat from core client for 30 sec - exiting 17:02:20 (123724): No heartbeat from core client for 30 sec - exiting 17:02:21 (123724): No heartbeat from core client for 30 sec - exiting 17:02:22 (123724): No heartbeat from core client for 30 sec - exiting 17:02:23 (123724): No heartbeat from core client for 30 sec - exiting 17:02:24 (123724): No heartbeat from core client for 30 sec - exiting 17:02:25 (123724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:58:07 (59936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:50:52 (380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:07 (3916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o2d1ko.pjm9c10 Error converting file to netcdf: dataout/o2d1ko.pim9c10 Error converting file to netcdf: dataout/o2d1ko.pfm9c10 Error converting file to netcdf: dataout/o2d1ka.phm9c10 Error converting file to netcdf: dataout/o2d1ka.pgm9c10 Error converting file to netcdf: dataout/o2d1ka.pem9c10 Error converting file to netcdf: dataout/o2d1ka.pdm9c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:23:53 (11792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:15:24 (1248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:11:48 (3244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:11:49 (3244): No heartbeat from core client for 30 sec - exiting 16:11:50 (3244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 22:38:36 (3520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:35:45 (8792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:01:36 (3784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16616, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16616, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16616, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16616, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish 07:15:08 (16616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17452, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17452, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Oct 2013 18:20:22	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	466,560	1,425,118	3.0545
23 Oct 2013 22:58:45	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	440,640	1,357,556	3.0809
23 Oct 2013 03:04:21	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	414,720	1,289,021	3.1082
22 Oct 2013 04:26:33	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	388,800	1,218,605	3.1343
20 Oct 2013 15:55:45	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	362,880	1,134,967	3.1277
18 Oct 2013 23:16:36	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	336,960	1,046,200	3.1048
17 Oct 2013 16:22:19	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	311,040	951,847	3.0602
16 Oct 2013 10:45:26	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	285,120	875,306	3.0700
15 Oct 2013 08:15:35	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	259,200	800,409	3.0880
14 Oct 2013 06:46:02	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	233,280	731,697	3.1366
12 Oct 2013 18:33:11	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	207,360	658,015	3.1733
11 Oct 2013 09:54:32	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	181,440	581,219	3.2034
10 Oct 2013 12:03:00	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	155,520	509,260	3.2746
09 Oct 2013 06:46:38	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	129,600	434,497	3.3526
08 Oct 2013 04:01:05	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	103,680	350,638	3.3819
06 Oct 2013 23:54:00	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	77,760	270,939	3.4843
05 Oct 2013 10:44:38	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	51,840	190,608	3.6769
03 Oct 2013 23:21:27	900317	16053406	hadcm3n_o2d1_2020_40_008376238_3	25,920	98,651	3.8060