Task 15425517

Name	hadcm3n_o53h_2060_40_008243997_0
Workunit	8399121
Created	2 Nov 2012, 19:07:13 UTC
Sent	2 Nov 2012, 19:07:43 UTC
Report deadline	2 Feb 2013, 2:34:54 UTC
Received	22 Dec 2012, 10:52:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	372855
Run time	27 days 18 hours 9 min 44 sec
CPU time	23 days 6 hours 54 min 42 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	0.89 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 17:58:43 (4012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:16 (4132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:18 (4132): No heartbeat from core client for 30 sec - exiting 18:05:19 (4132): No heartbeat from core client for 30 sec - exiting 18:05:21 (4132): No heartbeat from core client for 30 sec - exiting 18:05:22 (4132): No heartbeat from core client for 30 sec - exiting 18:05:23 (4132): No heartbeat from core client for 30 sec - exiting 18:05:25 (4132): No heartbeat from core client for 30 sec - exiting 18:05:26 (4132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 10:45:03 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:21:25 (6100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:21:26 (6100): No heartbeat from core client for 30 sec - exiting 21:52:43 (4412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:52:44 (4412): No heartbeat from core client for 30 sec - exiting 21:52:45 (4412): No heartbeat from core client for 30 sec - exiting 00:15:01 (4280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:55:00 (5260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:55:01 (5260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:26:12 (2500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:26:14 (2500): No heartbeat from core client for 30 sec - exiting 21:26:15 (2500): No heartbeat from core client for 30 sec - exiting 21:26:16 (2500): No heartbeat from core client for 30 sec - exiting 21:26:17 (2500): No heartbeat from core client for 30 sec - exiting 05:13:54 (8104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:13:55 (8104): No heartbeat from core client for 30 sec - exiting 12:11:27 (7176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:11:29 (7176): No heartbeat from core client for 30 sec - exiting 12:11:31 (7176): No heartbeat from core client for 30 sec - exiting 12:11:35 (7176): No heartbeat from core client for 30 sec - exiting 12:11:36 (7176): No heartbeat from core client for 30 sec - exiting 12:11:38 (7176): No heartbeat from core client for 30 sec - exiting 12:11:40 (7176): No heartbeat from core client for 30 sec - exiting 06:13:04 (3964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:13:08 (3964): No heartbeat from core client for 30 sec - exiting 06:13:09 (3964): No heartbeat from core client for 30 sec - exiting 06:13:10 (3964): No heartbeat from core client for 30 sec - exiting 06:13:11 (3964): No heartbeat from core client for 30 sec - exiting 06:13:13 (3964): No heartbeat from core client for 30 sec - exiting 06:13:14 (3964): No heartbeat from core client for 30 sec - exiting 06:13:15 (3964): No heartbeat from core client for 30 sec - exiting 06:13:16 (3964): No heartbeat from core client for 30 sec - exiting 06:13:18 (3964): No heartbeat from core client for 30 sec - exiting 06:13:19 (3964): No heartbeat from core client for 30 sec - exiting 06:13:20 (3964): No heartbeat from core client for 30 sec - exiting 06:13:21 (3964): No heartbeat from core client for 30 sec - exiting 06:13:23 (3964): No heartbeat from core client for 30 sec - exiting 06:13:24 (3964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 03:08:13 (524): No heartbeat from core client for 30 sec - exiting 03:08:14 (524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o53hko.pjq4c10 Error converting file to netcdf: dataout/o53hko.piq4c10 Error converting file to netcdf: dataout/o53hko.pfq4c10 Error converting file to netcdf: dataout/o53hka.phq4c10 Error converting file to netcdf: dataout/o53hka.pgq4c10 Error converting file to netcdf: dataout/o53hka.peq4c10 Error converting file to netcdf: dataout/o53hka.pdq4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:20:24 (5300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:35:29 (3884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:25:42 (3884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:48:03 (384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:48:05 (384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on o53hko.daq8410 CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:08:35 (5572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:49:01 (2540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=204, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=204, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=204, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=204, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=204, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=204, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Dec 2012 11:02:23	372855	15425517	hadcm3n_o53h_2060_40_008243997_0	233,280	1,864,005	7.9904
16 Dec 2012 17:06:33	372855	15425517	hadcm3n_o53h_2060_40_008243997_0	207,360	1,659,354	8.0023
14 Dec 2012 13:16:09	372855	15425517	hadcm3n_o53h_2060_40_008243997_0	181,440	1,453,485	8.0108
14 Dec 2012 13:16:09	372855	15425517	hadcm3n_o53h_2060_40_008243997_0	155,520	1,249,453	8.0340
05 Dec 2012 04:28:31	372855	15425517	hadcm3n_o53h_2060_40_008243997_0	129,600	1,043,001	8.0478
16 Nov 2012 08:50:51	372855	15425517	hadcm3n_o53h_2060_40_008243997_0	103,680	829,442	8.0000
13 Nov 2012 04:22:54	372855	15425517	hadcm3n_o53h_2060_40_008243997_0	77,760	624,268	8.0281
10 Nov 2012 08:00:47	372855	15425517	hadcm3n_o53h_2060_40_008243997_0	51,840	415,680	8.0185
07 Nov 2012 11:38:01	372855	15425517	hadcm3n_o53h_2060_40_008243997_0	25,920	208,479	8.0432