Task 14356729

Name	hadcm3n_o6n0_2020_40_007856162_1
Workunit	8011274
Created	4 Apr 2012, 15:28:17 UTC
Sent	4 Apr 2012, 15:28:25 UTC
Report deadline	4 Jul 2012, 22:55:36 UTC
Received	28 Apr 2012, 1:27:41 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1193410
Run time	12 days 14 hours 12 min 16 sec
CPU time	9 days 22 hours 30 min 52 sec
Validate state	Invalid
Credit	11,508.48
Device peak FLOPS	2.85 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9184, selfPID=9184, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o6n0ko.pjm8c10 CPDN Monitor - Quit request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7452, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:16:21 (7368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:16:23 (7368): No heartbeat from core client for 30 sec - exiting 12:16:24 (7368): No heartbeat from core client for 30 sec - exiting 12:16:25 (7368): No heartbeat from core client for 30 sec - exiting 12:16:26 (7368): No heartbeat from core client for 30 sec - exiting 12:16:27 (7368): No heartbeat from core client for 30 sec - exiting 12:16:28 (7368): No heartbeat from core client for 30 sec - exiting 12:16:29 (7368): No heartbeat from core client for 30 sec - exiting 12:16:30 (7368): No heartbeat from core client for 30 sec - exiting 12:16:31 (7368): No heartbeat from core client for 30 sec - exiting 12:16:32 (7368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Apr 2012 22:54:24	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	959,040	1,070,516	1.1162
26 Apr 2012 14:24:48	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	933,120	1,040,157	1.1147
26 Apr 2012 05:57:50	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	907,200	1,010,030	1.1133
25 Apr 2012 21:31:20	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	881,280	979,929	1.1119
25 Apr 2012 13:08:15	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	855,360	949,725	1.1103
25 Apr 2012 04:32:28	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	829,440	919,135	1.1081
24 Apr 2012 20:02:03	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	803,520	888,814	1.1062
24 Apr 2012 11:40:04	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	777,600	858,668	1.1043
24 Apr 2012 02:59:02	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	751,680	828,564	1.1023
23 Apr 2012 18:24:18	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	725,760	797,891	1.0994
23 Apr 2012 09:54:26	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	699,840	767,552	1.0968
23 Apr 2012 00:31:49	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	673,920	736,568	1.0930
22 Apr 2012 15:55:14	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	648,000	705,841	1.0893
22 Apr 2012 07:23:18	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	622,080	675,083	1.0852
21 Apr 2012 22:47:17	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	596,160	644,476	1.0810
21 Apr 2012 14:12:30	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	570,240	613,789	1.0764
21 Apr 2012 05:32:31	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	544,320	582,936	1.0709
20 Apr 2012 10:35:24	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	518,400	553,618	1.0679
19 Apr 2012 14:00:58	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	492,480	524,530	1.0651
18 Apr 2012 16:58:54	1193410	14356729	hadcm3n_o6n0_2020_40_007856162_1	466,560	495,089	1.0611