Task 13138756

Name	hadcm3n_o5ip_1940_40_007266268_2
Workunit	7464508
Created	15 Jul 2011, 15:11:42 UTC
Sent	15 Jul 2011, 15:13:07 UTC
Report deadline	14 Oct 2011, 22:40:18 UTC
Received	6 Sep 2011, 20:30:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1029371
Run time	20 days 11 hours 13 min 53 sec
CPU time	19 days 16 hours 29 min 17 sec
Validate state	Invalid
Credit	11,197.44
Device peak FLOPS	2.44 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.6</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:49:42 (3128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:11:49 (3684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o5ipko.pjg1c10 Error converting file to netcdf: dataout/o5ipko.pig1c10 Error converting file to netcdf: dataout/o5ipko.pfg1c10 Error converting file to netcdf: dataout/o5ipka.phg1c10 Error converting file to netcdf: dataout/o5ipka.pgg1c10 Error converting file to netcdf: dataout/o5ipka.peg1c10 Error converting file to netcdf: dataout/o5ipka.pdg1c10 Suspended CPDN Monitor - Suspend request from BOINC... 17:03:54 (4244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4908, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:35:58 (2716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:58:01 (3452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:58:02 (3452): No heartbeat from core client for 30 sec - exiting 22:58:03 (3452): No heartbeat from core client for 30 sec - exiting 22:58:04 (3452): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3760, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3760, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3760, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3760, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3760, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3760, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Sep 2011 03:06:04	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	933,120	1,700,871	1.8228
31 Aug 2011 11:55:53	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	907,200	1,655,361	1.8247
30 Aug 2011 23:28:33	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	881,280	1,610,852	1.8279
30 Aug 2011 10:40:20	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	855,360	1,566,675	1.8316
29 Aug 2011 21:52:14	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	829,440	1,521,789	1.8347
29 Aug 2011 09:05:25	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	803,520	1,476,456	1.8375
28 Aug 2011 01:58:08	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	777,600	1,430,557	1.8397
27 Aug 2011 08:22:37	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	751,680	1,386,142	1.8441
26 Aug 2011 19:28:01	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	725,760	1,340,699	1.8473
26 Aug 2011 06:36:27	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	699,840	1,295,350	1.8509
25 Aug 2011 15:13:49	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	673,920	1,247,689	1.8514
25 Aug 2011 00:49:42	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	648,000	1,199,878	1.8517
24 Aug 2011 08:44:09	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	622,080	1,151,381	1.8509
14 Aug 2011 14:55:40	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	596,160	1,104,567	1.8528
14 Aug 2011 01:01:17	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	570,240	1,057,250	1.8540
10 Aug 2011 08:19:28	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	544,320	1,007,199	1.8504
09 Aug 2011 16:01:40	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	518,400	956,569	1.8452
09 Aug 2011 00:55:28	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	492,480	905,798	1.8393
08 Aug 2011 11:02:26	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	466,560	856,386	1.8355
07 Aug 2011 21:11:48	1029371	13138756	hadcm3n_o5ip_1940_40_007266268_2	440,640	807,570	1.8327