Task 15526655

Name	hadcm3n_o71o_2140_40_008269966_3
Workunit	8425090
Created	9 Jan 2013, 1:03:54 UTC
Sent	9 Jan 2013, 1:03:56 UTC
Report deadline	10 Apr 2013, 8:31:07 UTC
Received	10 Apr 2013, 0:21:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1262675
Run time	44 days 6 hours 12 min 46 sec
CPU time	38 days 18 hours 59 min 9 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	1.30 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:45:19 (14796): No heartbeat from core client for 30 sec - exiting 19:45:20 (14796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:45:21 (14796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 20:38:25 (2708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6096, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:33:30 (3096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 01:40:34 PM No files match the supplied pattern. MainError: 01:40:34 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 02:27:04 AM No files match the supplied pattern. MainError: 02:27:04 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 08:21:52 PM No files match the supplied pattern. MainError: 08:21:52 PM No files match the supplied pattern. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3904, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 03:12:16 AM No files match the supplied pattern. MainError: 03:12:16 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2768, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2768, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2768, iMonCtr=1 Model crash detected, will try to restart... MainError: 01:35:53 PM No files match the supplied pattern. MainError: 01:35:53 PM No files match the supplied pattern. MainError: 01:32:12 PM No files match the supplied pattern. MainError: 01:32:12 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 05:09:14 AM No files match the supplied pattern. MainError: 05:09:14 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... 20:19:52 (3144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 08:19:16 PM No files match the supplied pattern. MainError: 08:19:16 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 12:47:20 AM No files match the supplied pattern. MainError: 12:47:20 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 10:42:19 AM No files match the supplied pattern. MainError: 10:42:19 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2712, iMonCtr=1 Model crash detected, will try to restart... Error converting file to netcdf: dataout/o71oka.ph11c10 Error converting file to netcdf: dataout/o71oka.pg11c10 Error converting file to netcdf: dataout/o71oka.pe11c10 MainError: 04:36:44 PM No files match the supplied pattern. MainError: 04:36:44 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Apr 2013 17:02:39	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	777,600	3,346,138	4.3032
05 Apr 2013 10:45:35	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	751,680	3,223,206	4.2880
03 Apr 2013 12:52:10	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	725,760	3,102,600	4.2750
25 Mar 2013 20:24:28	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	699,840	2,981,416	4.2601
21 Mar 2013 06:13:34	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	673,920	2,865,023	4.2513
18 Mar 2013 13:35:07	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	648,000	2,767,116	4.2702
17 Mar 2013 13:40:15	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	622,080	2,682,687	4.3124
14 Mar 2013 03:13:15	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	596,160	2,560,466	4.2949
01 Mar 2013 20:26:33	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	570,240	2,449,203	4.2950
18 Feb 2013 02:31:52	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	544,320	2,339,533	4.2981
16 Feb 2013 14:30:29	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	518,400	2,220,238	4.2829
14 Feb 2013 11:16:14	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	492,480	2,108,332	4.2811
12 Feb 2013 15:26:14	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	466,560	1,997,343	4.2810
07 Feb 2013 21:11:20	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	440,640	1,903,042	4.3188
06 Feb 2013 01:26:30	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	414,720	1,794,160	4.3262
04 Feb 2013 16:31:01	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	388,800	1,686,202	4.3369
03 Feb 2013 00:30:39	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	362,880	1,562,168	4.3049
01 Feb 2013 09:01:53	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	336,960	1,454,069	4.3153
31 Jan 2013 00:14:10	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	311,040	1,342,966	4.3177
28 Jan 2013 20:05:41	1262675	15526655	hadcm3n_o71o_2140_40_008269966_3	285,120	1,230,464	4.3156