Task 12460948

Name	famous_x8pc_599_200_007073111_2
Workunit	7276411
Created	4 Jan 2011, 1:30:20 UTC
Sent	4 Jan 2011, 1:33:12 UTC
Report deadline	5 Apr 2011, 9:00:23 UTC
Received	8 Jan 2011, 21:49:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1126344
Run time	1 days 9 hours 43 min 59 sec
CPU time	1 days 8 hours 35 min 26 sec
Validate state	Invalid
Credit	1,019.18
Device peak FLOPS	2.93 GFLOPS
Application version	UK Met Office FAMOUS v6.11 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... C16:45:06 (4700): No heartbeat from core client for 30 sec - exiting 16:45:07 (4700): No heartbeat from core client for 30 sec - exiting 16:45:08 (4700): No heartbeat from core client for 30 sec - exiting 16:45:09 (4700): No heartbeat from core client for 30 sec - exiting 16:45:10 (4700): No heartbeat from core client for 30 sec - exiting 16:45:11 (4700): No heartbeat from core client for 30 sec - exiting 16:45:12 (4700): No heartbeat from core client for 30 sec - exiting 16:45:13 (4700): No heartbeat from core client for 30 sec - exiting 16:45:14 (4700): No heartbeat from core client for 30 sec - exiting 16:45:15 (4700): No heartbeat from core client for 30 sec - exiting 16:45:16 (4700): No heartbeat from core client for 30 sec - exiting 16:45:17 (4700): No heartbeat from core client for 30 sec - exiting 16:45:18 (4700): No heartbeat from core client for 30 sec - exiting 16:45:19 (4700): No heartbeat from core client for 30 sec - exiting 16:45:20 (4700): No heartbeat from core client for 30 sec - exiting 16:45:21 (4700): No heartbeat from core client for 30 sec - exiting 16:45:22 (4700): No heartbeat from core client for 30 sec - exiting 16:45:23 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_x8pc_599_200_007073111\dataout\stdout_um.txt Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3092, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_x8pc_599_200_007073111\dataout\stdout_um.txt Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3092, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_x8pc_599_200_007073111\dataout\stdout_um.txt Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3092, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_x8pc_599_200_007073111\dataout\stdout_um.txt Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3092, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_x8pc_599_200_007073111\dataout\stdout_um.txt Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3092, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_x8pc_599_200_007073111\dataout\stdout_um.txt Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3092, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2720, selfPID=2720, iMonCtr=1 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/ocean_restart.day after 11 attempts Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/ocean_restart.day after 11 attempts Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/ocean_restart.day after 11 attempts CPDN Monitor - Quit request from BOINC... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/ocean_restart.day after 11 attempts Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/ocean_restart.day after 11 attempts Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/ocean_restart.day after 11 attempts CPDN Monitor - Quit request from BOINC... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/ocean_restart.day after 11 attempts Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_x8pc_599_200_007073111/dataout/ocean_restart.day after 11 attempts Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Sorry, too many model crashes! :-( 16:47:55 (2084): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Jan 2011 20:48:34	1126344	12460948	famous_x8pc_599_200_007073111_2	308,906	114,612	0.3710
08 Jan 2011 19:10:22	1126344	12460948	famous_x8pc_599_200_007073111_2	299,546	111,185	0.3712
08 Jan 2011 18:42:41	1126344	12460948	famous_x8pc_599_200_007073111_2	290,186	107,746	0.3713
08 Jan 2011 16:50:34	1126344	12460948	famous_x8pc_599_200_007073111_2	280,826	104,344	0.3716
08 Jan 2011 15:53:00	1126344	12460948	famous_x8pc_599_200_007073111_2	271,466	100,953	0.3719
07 Jan 2011 23:49:17	1126344	12460948	famous_x8pc_599_200_007073111_2	262,106	97,553	0.3722
05 Jan 2011 07:39:54	1126344	12460948	famous_x8pc_599_200_007073111_2	252,746	94,118	0.3724
05 Jan 2011 06:14:33	1126344	12460948	famous_x8pc_599_200_007073111_2	243,386	90,691	0.3726
05 Jan 2011 04:22:40	1126344	12460948	famous_x8pc_599_200_007073111_2	234,026	87,250	0.3728
05 Jan 2011 03:25:46	1126344	12460948	famous_x8pc_599_200_007073111_2	224,666	83,749	0.3728
05 Jan 2011 02:23:42	1126344	12460948	famous_x8pc_599_200_007073111_2	215,306	80,268	0.3728
05 Jan 2011 00:08:42	1126344	12460948	famous_x8pc_599_200_007073111_2	205,946	76,790	0.3729
04 Jan 2011 23:09:31	1126344	12460948	famous_x8pc_599_200_007073111_2	196,586	73,295	0.3728
04 Jan 2011 22:00:11	1126344	12460948	famous_x8pc_599_200_007073111_2	187,226	69,854	0.3731
04 Jan 2011 21:03:06	1126344	12460948	famous_x8pc_599_200_007073111_2	177,866	66,373	0.3732
04 Jan 2011 20:03:09	1126344	12460948	famous_x8pc_599_200_007073111_2	168,506	62,872	0.3731
04 Jan 2011 19:03:08	1126344	12460948	famous_x8pc_599_200_007073111_2	159,146	59,388	0.3732
04 Jan 2011 18:42:23	1126344	12460948	famous_x8pc_599_200_007073111_2	149,786	55,886	0.3731
04 Jan 2011 17:02:25	1126344	12460948	famous_x8pc_599_200_007073111_2	140,426	52,385	0.3730
04 Jan 2011 16:03:54	1126344	12460948	famous_x8pc_599_200_007073111_2	131,066	48,904	0.3731