Task 15800516

Name	hadcm3n_ziyo_1960_40_008366205_1
Workunit	8517064
Created	29 May 2013, 8:28:44 UTC
Sent	29 May 2013, 8:28:48 UTC
Report deadline	28 Aug 2013, 15:55:59 UTC
Received	21 Jun 2013, 1:24:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1237173
Run time	1 days 21 hours 44 min 47 sec
CPU time	1 days 8 hours 28 min 41 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	3.34 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 16:11:25 (24716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:11:27 (24716): No heartbeat from core client for 30 sec - exiting 16:11:28 (24716): No heartbeat from core client for 30 sec - exiting 16:11:29 (24716): No heartbeat from core client for 30 sec - exiting 16:11:30 (24716): No heartbeat from core client for 30 sec - exiting 16:11:31 (24716): No heartbeat from core client for 30 sec - exiting 16:11:32 (24716): No heartbeat from core client for 30 sec - exiting 16:11:33 (24716): No heartbeat from core client for 30 sec - exiting 16:11:34 (24716): No heartbeat from core client for 30 sec - exiting 16:11:35 (24716): No heartbeat from core client for 30 sec - exiting 16:11:36 (24716): No heartbeat from core client for 30 sec - exiting 17:42:44 (22020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:34:06 (26536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 20:13:45 (26652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:13:46 (26652): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... 00:45:16 (27864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:41:57 (28412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:42:02 (28412): No heartbeat from core client for 30 sec - exiting 06:57:44 (26724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:26:56 (28788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:07 (30948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:11 (30948): No heartbeat from core client for 30 sec - exiting 07:52:12 (30948): No heartbeat from core client for 30 sec - exiting 07:52:13 (30948): No heartbeat from core client for 30 sec - exiting 07:52:14 (30948): No heartbeat from core client for 30 sec - exiting 07:52:15 (30948): No heartbeat from core client for 30 sec - exiting 07:52:16 (30948): No heartbeat from core client for 30 sec - exiting 07:52:17 (30948): No heartbeat from core client for 30 sec - exiting 07:52:18 (30948): No heartbeat from core client for 30 sec - exiting 07:52:19 (30948): No heartbeat from core client for 30 sec - exiting 07:52:20 (30948): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 08:09:15 (30032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:16 (30032): No heartbeat from core client for 30 sec - exiting 10:06:41 (26588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:43 (26588): No heartbeat from core client for 30 sec - exiting 10:10:46 (31188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:36 (29572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:32:58 (31316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:12:39 (32084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:02 (33108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:20:54 (33224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:56:35 (6340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:07:09 (9940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:07:10 (9940): No heartbeat from core client for 30 sec - exiting 22:07:11 (9940): No heartbeat from core client for 30 sec - exiting 22:07:12 (9940): No heartbeat from core client for 30 sec - exiting 06:36:51 (11804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:41:17 (11200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:41:19 (11200): No heartbeat from core client for 30 sec - exiting 06:41:20 (11200): No heartbeat from core client for 30 sec - exiting 07:09:21 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:55 (10564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:46:59 (3148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:03 (3148): No heartbeat from core client for 30 sec - exiting 08:47:04 (3148): No heartbeat from core client for 30 sec - exiting 08:47:05 (3148): No heartbeat from core client for 30 sec - exiting 09:01:09 (10836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:01:11 (10836): No heartbeat from core client for 30 sec - exiting 09:13:14 (9508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:22 (9508): No heartbeat from core client for 30 sec - exiting 09:13:23 (9508): No heartbeat from core client for 30 sec - exiting 09:13:24 (9508): No heartbeat from core client for 30 sec - exiting 09:13:25 (9508): No heartbeat from core client for 30 sec - exiting 09:13:26 (9508): No heartbeat from core client for 30 sec - exiting 09:13:27 (9508): No heartbeat from core client for 30 sec - exiting 09:13:28 (9508): No heartbeat from core client for 30 sec - exiting 09:13:29 (9508): No heartbeat from core client for 30 sec - exiting 09:19:53 (4204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:20:02 (4204): No heartbeat from core client for 30 sec - exiting 09:20:03 (4204): No heartbeat from core client for 30 sec - exiting 09:20:04 (4204): No heartbeat from core client for 30 sec - exiting 09:20:05 (4204): No heartbeat from core client for 30 sec - exiting 09:20:06 (4204): No heartbeat from core client for 30 sec - exiting 09:20:07 (4204): No heartbeat from core client for 30 sec - exiting 09:20:08 (4204): No heartbeat from core client for 30 sec - exiting 09:20:09 (4204): No heartbeat from core client for 30 sec - exiting 09:20:10 (4204): No heartbeat from core client for 30 sec - exiting 09:20:11 (4204): No heartbeat from core client for 30 sec - exiting 11:07:02 (4844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:17 (12328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:21 (12328): No heartbeat from core client for 30 sec - exiting 11:50:22 (12328): No heartbeat from core client for 30 sec - exiting 11:50:23 (12328): No heartbeat from core client for 30 sec - exiting 11:50:24 (12328): No heartbeat from core client for 30 sec - exiting 11:50:25 (12328): No heartbeat from core client for 30 sec - exiting 11:50:26 (12328): No heartbeat from core client for 30 sec - exiting 11:50:27 (12328): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 12:28:29 (5444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:36:00 (13652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:36:12 (13652): No heartbeat from core client for 30 sec - exiting 12:36:13 (13652): No heartbeat from core client for 30 sec - exiting 12:36:14 (13652): No heartbeat from core client for 30 sec - exiting 12:36:15 (13652): No heartbeat from core client for 30 sec - exiting 12:36:16 (13652): No heartbeat from core client for 30 sec - exiting 12:36:17 (13652): No heartbeat from core client for 30 sec - exiting 12:36:18 (13652): No heartbeat from core client for 30 sec - exiting 12:36:19 (13652): No heartbeat from core client for 30 sec - exiting 12:36:20 (13652): No heartbeat from core client for 30 sec - exiting 12:36:21 (13652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 12:57:57 (6236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:17:46 (1588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:43:42 (13036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:13:44 (14560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:43:05 (6520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:43:26 (6520): No heartbeat from core client for 30 sec - exiting 16:16:41 (13440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:46 (14692): No heartbeat from core client for 30 sec - exiting 16:55:48 (14692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ziyoko.pjg4c10 Error converting file to netcdf: dataout/ziyoko.pig4c10 Error converting file to netcdf: dataout/ziyoko.pfg4c10 Error converting file to netcdf: dataout/ziyoka.phg4c10 Error converting file to netcdf: dataout/ziyoka.pgg4c10 Error converting file to netcdf: dataout/ziyoka.peg4c10 Error converting file to netcdf: dataout/ziyoka.pdg4c10 17:53:39 (16264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:53:40 (16264): No heartbeat from core client for 30 sec - exiting 17:53:41 (16264): No heartbeat from core client for 30 sec - exiting 17:53:42 (16264): No heartbeat from core client for 30 sec - exiting 17:53:43 (16264): No heartbeat from core client for 30 sec - exiting 17:53:44 (16264): No heartbeat from core client for 30 sec - exiting 17:53:45 (16264): No heartbeat from core client for 30 sec - exiting 17:53:46 (16264): No heartbeat from core client for 30 sec - exiting 17:53:47 (16264): No heartbeat from core client for 30 sec - exiting 17:53:48 (16264): No heartbeat from core client for 30 sec - exiting 17:53:49 (16264): No heartbeat from core client for 30 sec - exiting 18:30:57 (14232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:00 (14232): No heartbeat from core client for 30 sec - exiting 18:31:01 (14232): No heartbeat from core client for 30 sec - exiting 18:31:02 (14232): No heartbeat from core client for 30 sec - exiting 18:31:03 (14232): No heartbeat from core client for 30 sec - exiting 18:31:04 (14232): No heartbeat from core client for 30 sec - exiting 19:27:03 (16104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14752, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14752, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14752, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14752, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14752, iMonCtr=1 Model crash detected, will try to restart... 21:13:32 (14752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:13:33 (14752): No heartbeat from core client for 30 sec - exiting 21:13:34 (14752): No heartbeat from core client for 30 sec - exiting 21:13:35 (14752): No heartbeat from core client for 30 sec - exiting 21:13:36 (14752): No heartbeat from core client for 30 sec - exiting 21:13:37 (14752): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17736, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Jun 2013 21:41:55	1237173	15800516	hadcm3n_ziyo_1960_40_008366205_1	103,680	112,006	1.0803
20 Jun 2013 08:58:01	1237173	15800516	hadcm3n_ziyo_1960_40_008366205_1	77,760	83,035	1.0678
19 Jun 2013 19:24:08	1237173	15800516	hadcm3n_ziyo_1960_40_008366205_1	51,840	56,883	1.0973
19 Jun 2013 08:41:20	1237173	15800516	hadcm3n_ziyo_1960_40_008366205_1	25,920	28,813	1.1116