Task 15658252

Name	hadcm3n_o0ti_2140_40_008268855_2
Workunit	8423979
Created	11 Mar 2013, 20:40:49 UTC
Sent	11 Mar 2013, 20:41:13 UTC
Report deadline	11 Jun 2013, 4:08:24 UTC
Received	23 Apr 2013, 19:09:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1227307
Run time	8 days 5 hours 0 min 5 sec
CPU time	7 days 15 hours 31 min 56 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.43 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5912, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5820, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4120, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6088, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5956, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5372, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5900, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3888, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5592, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4476, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5624, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5768, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5612, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3704, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 11:59:52 AM No files match the supplied pattern. MainError: 11:59:52 AM No files match the supplied pattern. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5132, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5368, iMonCtr=1 Model crash detected, will try to restart... MainError: 09:51:38 AM No files match the supplied pattern. MainError: 09:51:38 AM No files match the supplied pattern. MainError: 04:12:44 PM No files match the supplied pattern. MainError: 04:12:44 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 08:39:31 PM No files match the supplied pattern. MainError: 08:39:31 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5840, iMonCtr=1 Model crash detected, will try to restart... MainError: 07:24:08 PM No files match the supplied pattern. MainError: 07:24:08 PM No files match the supplied pattern. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5828, iMonCtr=1 Model crash detected, will try to restart... MainError: 06:29:38 PM No files match the supplied pattern. MainError: 06:29:39 PM No files match the supplied pattern. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5884, iMonCtr=1 Model crash detected, will try to restart... MainError: 07:05:50 PM No files match the supplied pattern. MainError: 07:05:50 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 06:49:49 PM No files match the supplied pattern. MainError: 06:49:49 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 09:57:38 PM No files match the supplied pattern. MainError: 09:57:38 PM No files match the supplied pattern. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4460, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 08:54:33 AM No files match the supplied pattern. MainError: 08:54:33 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Error converting file to netcdf: dataout/o0tika.ph11c10 Error converting file to netcdf: dataout/o0tika.pg11c10 Error converting file to netcdf: dataout/o0tika.pe11c10 MainError: 07:43:06 PM No files match the supplied pattern. MainError: 07:43:06 PM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Apr 2013 19:47:21	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	777,600	661,928	0.8512
20 Apr 2013 08:58:59	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	751,680	639,749	0.8511
16 Apr 2013 22:02:59	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	725,760	618,160	0.8517
15 Apr 2013 18:58:26	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	699,840	596,811	0.8528
14 Apr 2013 19:59:59	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	673,920	575,131	0.8534
13 Apr 2013 19:25:15	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	648,000	553,329	0.8539
11 Apr 2013 20:23:47	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	622,080	531,632	0.8546
08 Apr 2013 21:35:36	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	596,160	509,771	0.8551
07 Apr 2013 16:36:07	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	570,240	488,006	0.8558
07 Apr 2013 10:03:17	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	544,320	466,367	0.8568
06 Apr 2013 12:55:09	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	518,400	444,783	0.8580
05 Apr 2013 20:41:20	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	492,480	422,891	0.8587
01 Apr 2013 22:54:51	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	466,560	401,550	0.8607
01 Apr 2013 17:52:50	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	440,640	380,478	0.8635
01 Apr 2013 10:40:02	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	414,720	359,109	0.8659
30 Mar 2013 21:40:40	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	388,800	336,564	0.8656
29 Mar 2013 22:20:38	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	362,880	313,938	0.8651
26 Mar 2013 21:30:41	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	336,960	291,269	0.8644
24 Mar 2013 19:24:52	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	311,040	268,729	0.8640
24 Mar 2013 09:17:15	1227307	15658252	hadcm3n_o0ti_2140_40_008268855_2	285,120	246,143	0.8633