Task 15767114

Name	hadcm3n_4kcq_1980_40_008360438_0
Workunit	8511297
Created	9 May 2013, 10:12:36 UTC
Sent	9 May 2013, 10:12:48 UTC
Report deadline	8 Aug 2013, 17:39:59 UTC
Received	24 Jul 2013, 7:23:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1193410
Run time	12 days 14 hours 5 min 3 sec
CPU time	12 days 9 hours 28 min 50 sec
Validate state	Invalid
Credit	11,197.44
Device peak FLOPS	2.83 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 21:23:01 (6904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:02:07 (6984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12980, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 09:40:03 (7024): No heartbeat from core client for 30 sec - exiting 09:40:04 (7024): No heartbeat from core client for 30 sec - exiting 09:40:05 (7024): No heartbeat from core client for 30 sec - exiting 09:40:06 (7024): No heartbeat from core client for 30 sec - exiting 09:40:07 (7024): No heartbeat from core client for 30 sec - exiting 09:40:08 (7024): No heartbeat from core client for 30 sec - exiting 09:40:09 (7024): No heartbeat from core client for 30 sec - exiting 09:40:10 (7024): No heartbeat from core client for 30 sec - exiting 09:40:11 (7024): No heartbeat from core client for 30 sec - exiting 09:40:12 (7024): No heartbeat from core client for 30 sec - exiting 09:40:13 (7024): No heartbeat from core client for 30 sec - exiting 09:40:14 (7024): No heartbeat from core client for 30 sec - exiting 09:40:15 (7024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4kcqko.pji2c10 Error converting file to netcdf: dataout/4kcqko.pii2c10 Error converting file to netcdf: dataout/4kcqko.pfi2c10 Error converting file to netcdf: dataout/4kcqka.phi2c10 Error converting file to netcdf: dataout/4kcqka.pgi2c10 Error converting file to netcdf: dataout/4kcqka.pei2c10 Error converting file to netcdf: dataout/4kcqka.pdi2c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:48:52 (8940): No heartbeat from core client for 30 sec - exiting 15:48:53 (8940): No heartbeat from core client for 30 sec - exiting 15:48:54 (8940): No heartbeat from core client for 30 sec - exiting 15:48:55 (8940): No heartbeat from core client for 30 sec - exiting 15:48:56 (8940): No heartbeat from core client for 30 sec - exiting 15:48:57 (8940): No heartbeat from core client for 30 sec - exiting 15:48:58 (8940): No heartbeat from core client for 30 sec - exiting 15:48:59 (8940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:49:00 (8940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:11:54 (16060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:10:56 (12980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:32:40 (6752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:33:15 (6692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:03:41 (11584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:40:08 (10924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:08:30 (9260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:42:20 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:38:32 (4076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:38:33 (4076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15308, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:28:45 (5780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:17:38 (3332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3704, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3704, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5940, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5940, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5940, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5940, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Jul 2013 22:05:47	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	933,120	1,064,849	1.1412
23 Jul 2013 20:15:34	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	907,200	1,036,174	1.1422
23 Jul 2013 19:34:17	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	881,280	1,007,895	1.1437
23 Jul 2013 17:16:00	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	855,360	980,070	1.1458
23 Jul 2013 17:15:59	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	829,440	954,189	1.1504
04 Jul 2013 14:28:19	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	803,520	924,543	1.1506
03 Jul 2013 13:12:35	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	777,600	894,451	1.1503
02 Jul 2013 12:33:55	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	751,680	864,393	1.1499
02 Jul 2013 11:26:51	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	725,760	832,183	1.1466
28 Jun 2013 08:33:21	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	699,840	798,967	1.1416
27 Jun 2013 09:04:06	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	673,920	768,451	1.1403
25 Jun 2013 14:24:11	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	648,000	736,402	1.1364
25 Jun 2013 06:15:06	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	622,080	706,387	1.1355
20 Jun 2013 14:55:48	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	596,160	675,159	1.1325
19 Jun 2013 14:54:06	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	570,240	643,734	1.1289
18 Jun 2013 14:43:34	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	544,320	614,956	1.1298
14 Jun 2013 10:53:29	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	518,400	586,912	1.1322
12 Jun 2013 15:01:32	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	492,480	557,719	1.1325
11 Jun 2013 15:11:05	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	466,560	528,593	1.1330
10 Jun 2013 14:50:19	1193410	15767114	hadcm3n_4kcq_1980_40_008360438_0	440,640	500,326	1.1355