Task 15552126

Name	hadcm3n_3kp6_1980_40_008283585_1
Workunit	8434720
Created	19 Jan 2013, 17:58:54 UTC
Sent	19 Jan 2013, 17:58:59 UTC
Report deadline	21 Apr 2013, 1:26:10 UTC
Received	9 Apr 2013, 19:15:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1263465
Run time	8 days 2 hours 53 min 50 sec
CPU time	6 days 17 hours 57 min
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	1.72 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4660, iMonCtr=1 Model crash detected, will try to restart... 22:34:35 (3928): No heartbeat from core client for 30 sec - exiting 22:34:36 (3928): No heartbeat from core client for 30 sec - exiting 22:34:37 (3928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4676, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:26:07 (4232): No heartbeat from core client for 30 sec - exiting 18:26:08 (4232): No heartbeat from core client for 30 sec - exiting 18:26:09 (4232): No heartbeat from core client for 30 sec - exiting 18:26:10 (4232): No heartbeat from core client for 30 sec - exiting 18:26:11 (4232): No heartbeat from core client for 30 sec - exiting 18:26:13 (4232): No heartbeat from core client for 30 sec - exiting 18:26:14 (4232): No heartbeat from core client for 30 sec - exiting 18:26:15 (4232): No heartbeat from core client for 30 sec - exiting 18:26:16 (4232): No heartbeat from core client for 30 sec - exiting 18:26:17 (4232): No heartbeat from core client for 30 sec - exiting 18:26:18 (4232): No heartbeat from core client for 30 sec - exiting 18:26:19 (4232): No heartbeat from core client for 30 sec - exiting 18:26:20 (4232): No heartbeat from core client for 30 sec - exiting 18:26:21 (4232): No heartbeat from core client for 30 sec - exiting 18:26:22 (4232): No heartbeat from core client for 30 sec - exiting 18:26:23 (4232): No heartbeat from core client for 30 sec - exiting 18:26:25 (4232): No heartbeat from core client for 30 sec - exiting 18:26:26 (4232): No heartbeat from core client for 30 sec - exiting 18:26:27 (4232): No heartbeat from core client for 30 sec - exiting 18:26:28 (4232): No heartbeat from core client for 30 sec - exiting 18:26:29 (4232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1492, iMonCtr=1 Model crash detected, will try to restart... 17:25:56 (3016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:22:03 (4900): No heartbeat from core client for 30 sec - exiting 18:22:04 (4900): No heartbeat from core client for 30 sec - exiting 18:22:05 (4900): No heartbeat from core client for 30 sec - exiting 18:22:06 (4900): No heartbeat from core client for 30 sec - exiting 18:22:07 (4900): No heartbeat from core client for 30 sec - exiting 18:22:08 (4900): No heartbeat from core client for 30 sec - exiting 18:22:10 (4900): No heartbeat from core client for 30 sec - exiting 18:22:11 (4900): No heartbeat from core client for 30 sec - exiting 18:22:12 (4900): No heartbeat from core client for 30 sec - exiting 18:22:13 (4900): No heartbeat from core client for 30 sec - exiting 18:22:14 (4900): No heartbeat from core client for 30 sec - exiting 18:22:15 (4900): No heartbeat from core client for 30 sec - exiting 18:22:16 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:38:18 (3252): No heartbeat from core client for 30 sec - exiting 20:38:19 (3252): No heartbeat from core client for 30 sec - exiting 20:38:21 (3252): No heartbeat from core client for 30 sec - exiting 20:38:22 (3252): No heartbeat from core client for 30 sec - exiting 20:38:23 (3252): No heartbeat from core client for 30 sec - exiting 20:38:24 (3252): No heartbeat from core client for 30 sec - exiting 20:38:25 (3252): No heartbeat from core client for 30 sec - exiting 20:38:26 (3252): No heartbeat from core client for 30 sec - exiting 20:38:27 (3252): No heartbeat from core client for 30 sec - exiting 20:38:28 (3252): No heartbeat from core client for 30 sec - exiting 20:38:29 (3252): No heartbeat from core client for 30 sec - exiting 20:38:30 (3252): No heartbeat from core client for 30 sec - exiting 20:38:32 (3252): No heartbeat from core client for 30 sec - exiting 20:38:33 (3252): No heartbeat from core client for 30 sec - exiting 20:38:34 (3252): No heartbeat from core client for 30 sec - exiting 20:38:35 (3252): No heartbeat from core client for 30 sec - exiting 20:38:36 (3252): No heartbeat from core client for 30 sec - exiting 20:38:37 (3252): No heartbeat from core client for 30 sec - exiting 20:38:39 (3252): No heartbeat from core client for 30 sec - exiting 20:38:40 (3252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3kp6ko.pji6c10 Error converting file to netcdf: dataout/3kp6ko.pii6c10 Error converting file to netcdf: dataout/3kp6ko.pfi6c10 Error converting file to netcdf: dataout/3kp6ka.phi6c10 Error converting file to netcdf: dataout/3kp6ka.pgi6c10 Error converting file to netcdf: dataout/3kp6ka.pei6c10 Error converting file to netcdf: dataout/3kp6ka.pdi6c10 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1596, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:01:02 (4708): No heartbeat from core client for 30 sec - exiting 11:01:04 (4708): No heartbeat from core client for 30 sec - exiting 11:01:05 (4708): No heartbeat from core client for 30 sec - exiting 11:01:06 (4708): No heartbeat from core client for 30 sec - exiting 11:01:07 (4708): No heartbeat from core client for 30 sec - exiting 11:01:08 (4708): No heartbeat from core client for 30 sec - exiting 11:01:09 (4708): No heartbeat from core client for 30 sec - exiting 11:01:10 (4708): No heartbeat from core client for 30 sec - exiting 11:01:11 (4708): No heartbeat from core client for 30 sec - exiting 11:01:12 (4708): No heartbeat from core client for 30 sec - exiting 11:01:13 (4708): No heartbeat from core client for 30 sec - exiting 11:01:14 (4708): No heartbeat from core client for 30 sec - exiting 11:01:16 (4708): No heartbeat from core client for 30 sec - exiting 11:01:17 (4708): No heartbeat from core client for 30 sec - exiting 11:01:18 (4708): No heartbeat from core client for 30 sec - exiting 11:01:19 (4708): No heartbeat from core client for 30 sec - exiting 11:01:20 (4708): No heartbeat from core client for 30 sec - exiting 11:01:21 (4708): No heartbeat from core client for 30 sec - exiting 11:01:22 (4708): No heartbeat from core client for 30 sec - exiting 11:01:23 (4708): No heartbeat from core client for 30 sec - exiting 11:01:24 (4708): No heartbeat from core client for 30 sec - exiting 11:01:25 (4708): No heartbeat from core client for 30 sec - exiting 11:01:26 (4708): No heartbeat from core client for 30 sec - exiting 11:01:28 (4708): No heartbeat from core client for 30 sec - exiting 11:01:29 (4708): No heartbeat from core client for 30 sec - exiting 11:01:30 (4708): No heartbeat from core client for 30 sec - exiting 11:01:31 (4708): No heartbeat from core client for 30 sec - exiting 11:01:32 (4708): No heartbeat from core client for 30 sec - exiting 11:01:33 (4708): No heartbeat from core client for 30 sec - exiting 11:01:34 (4708): No heartbeat from core client for 30 sec - exiting 11:01:35 (4708): No heartbeat from core client for 30 sec - exiting 11:01:36 (4708): No heartbeat from core client for 30 sec - exiting 11:01:38 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:52:13 (3644): No heartbeat from core client for 30 sec - exiting 18:52:14 (3644): No heartbeat from core client for 30 sec - exiting 18:52:15 (3644): No heartbeat from core client for 30 sec - exiting 18:52:17 (3644): No heartbeat from core client for 30 sec - exiting 18:52:18 (3644): No heartbeat from core client for 30 sec - exiting 18:52:19 (3644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5196, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5196, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5196, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 10:27:39 (5292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5708, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4148, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4148, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Mar 2013 20:01:48	1263465	15552126	hadcm3n_3kp6_1980_40_008283585_1	207,360	524,901	2.5314
20 Mar 2013 21:00:11	1263465	15552126	hadcm3n_3kp6_1980_40_008283585_1	181,440	459,424	2.5321
16 Mar 2013 16:45:59	1263465	15552126	hadcm3n_3kp6_1980_40_008283585_1	155,520	395,032	2.5401
09 Mar 2013 01:45:42	1263465	15552126	hadcm3n_3kp6_1980_40_008283585_1	129,600	330,121	2.5472
03 Mar 2013 19:39:59	1263465	15552126	hadcm3n_3kp6_1980_40_008283585_1	103,680	265,712	2.5628
25 Feb 2013 23:32:01	1263465	15552126	hadcm3n_3kp6_1980_40_008283585_1	77,760	200,052	2.5727
29 Jan 2013 22:21:48	1263465	15552126	hadcm3n_3kp6_1980_40_008283585_1	51,840	133,025	2.5661
25 Jan 2013 22:28:02	1263465	15552126	hadcm3n_3kp6_1980_40_008283585_1	25,920	66,717	2.5740