Task 11671206

Name	famous_ugrz_1799_200_006654586_5
Workunit	6857958
Created	23 Aug 2010, 9:06:02 UTC
Sent	23 Aug 2010, 9:30:58 UTC
Report deadline	22 Nov 2010, 16:58:09 UTC
Received	26 Aug 2010, 16:09:33 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1057648
Run time	2 days 11 hours 40 min 31 sec
CPU time	2 days 10 hours 45 min 2 sec
Validate state	Invalid
Credit	2,316.21
Device peak FLOPS	3.05 GFLOPS
Application version	UK Met Office FAMOUS v6.11 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 13:08:48 (2464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:08:49 (2464): No heartbeat from core client for 30 sec - exiting 15:03:25 (4324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:08:51 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:09:50 (2668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:13:17 (3992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:13:53 (1720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:55:01 (2876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:02 (2876): No heartbeat from core client for 30 sec - exiting 16:55:03 (2876): No heartbeat from core client for 30 sec - exiting 16:55:04 (2876): No heartbeat from core client for 30 sec - exiting 16:55:05 (2876): No heartbeat from core client for 30 sec - exiting 16:55:06 (2876): No heartbeat from core client for 30 sec - exiting 16:55:07 (2876): No heartbeat from core client for 30 sec - exiting 16:55:08 (2876): No heartbeat from core client for 30 sec - exiting 16:55:09 (2876): No heartbeat from core client for 30 sec - exiting 16:55:10 (2876): No heartbeat from core client for 30 sec - exiting 16:55:11 (2876): No heartbeat from core client for 30 sec - exiting 16:57:47 (2060): No heartbeat from core client for 30 sec - exiting 16:57:51 (2060): No heartbeat from core client for 30 sec - exiting 16:57:52 (2060): No heartbeat from core client for 30 sec - exiting 16:57:53 (2060): No heartbeat from core client for 30 sec - exiting 16:57:54 (2060): No heartbeat from core client for 30 sec - exiting 16:57:55 (2060): No heartbeat from core client for 30 sec - exiting 16:57:56 (2060): No heartbeat from core client for 30 sec - exiting 16:57:57 (2060): No heartbeat from core client for 30 sec - exiting 16:57:58 (2060): No heartbeat from core client for 30 sec - exiting 16:57:59 (2060): No heartbeat from core client for 30 sec - exiting 16:58:00 (2060): No heartbeat from core client for 30 sec - exiting 16:58:01 (2060): No heartbeat from core client for 30 sec - exiting 16:58:02 (2060): No heartbeat from core client for 30 sec - exiting 16:58:33 (2060): No heartbeat from core client for 30 sec - exiting 16:58:34 (2060): No heartbeat from core client for 30 sec - exiting 16:58:35 (2060): No heartbeat from core client for 30 sec - exiting 16:58:36 (2060): No heartbeat from core client for 30 sec - exiting 16:58:37 (2060): No heartbeat from core client for 30 sec - exiting 16:58:38 (2060): No heartbeat from core client for 30 sec - exiting 16:58:39 (2060): No heartbeat from core client for 30 sec - exiting 16:58:40 (2060): No heartbeat from core client for 30 sec - exiting 16:58:41 (2060): No heartbeat from core client for 30 sec - exiting 16:58:42 (2060): No heartbeat from core client for 30 sec - exiting 16:58:43 (2060): No heartbeat from core client for 30 sec - exiting 16:58:44 (2060): No heartbeat from core client for 30 sec - exiting 16:58:45 (2060): No heartbeat from core client for 30 sec - exiting 16:58:46 (2060): No heartbeat from core client for 30 sec - exiting 16:58:47 (2060): No heartbeat from core client for 30 sec - exiting 16:58:48 (2060): No heartbeat from core client for 30 sec - exiting 16:58:49 (2060): No heartbeat from core client for 30 sec - exiting 16:58:50 (2060): No heartbeat from core client for 30 sec - exiting 16:58:51 (2060): No heartbeat from core client for 30 sec - exiting 16:58:52 (2060): No heartbeat from core client for 30 sec - exiting 16:58:53 (2060): No heartbeat from core client for 30 sec - exiting 16:58:54 (2060): No heartbeat from core client for 30 sec - exiting 16:58:55 (2060): No heartbeat from core client for 30 sec - exiting 16:58:56 (2060): No heartbeat from core client for 30 sec - exiting 16:58:57 (2060): No heartbeat from core client for 30 sec - exiting 16:58:58 (2060): No heartbeat from core client for 30 sec - exiting 16:58:59 (2060): No heartbeat from core client for 30 sec - exiting 16:59:00 (2060): No heartbeat from core client for 30 sec - exiting 16:59:01 (2060): No heartbeat from core client for 30 sec - exiting 16:59:02 (2060): No heartbeat from core client for 30 sec - exiting 16:59:03 (2060): No heartbeat from core client for 30 sec - exiting 16:59:04 (2060): No heartbeat from core client for 30 sec - exiting 16:59:05 (2060): No heartbeat from core client for 30 sec - exiting 16:59:06 (2060): No heartbeat from core client for 30 sec - exiting 16:59:07 (2060): No heartbeat from core client for 30 sec - exiting 16:59:08 (2060): No heartbeat from core client for 30 sec - exiting 16:59:09 (2060): No heartbeat from core client for 30 sec - exiting 16:59:10 (2060): No heartbeat from core client for 30 sec - exiting 16:59:11 (2060): No heartbeat from core client for 30 sec - exiting 16:59:12 (2060): No heartbeat from core client for 30 sec - exiting 16:59:13 (2060): No heartbeat from core client for 30 sec - exiting 16:59:14 (2060): No heartbeat from core client for 30 sec - exiting 16:59:15 (2060): No heartbeat from core client for 30 sec - exiting 16:59:16 (2060): No heartbeat from core client for 30 sec - exiting 16:59:17 (2060): No heartbeat from core client for 30 sec - exiting 16:59:18 (2060): No heartbeat from core client for 30 sec - exiting 16:59:19 (2060): No heartbeat from core client for 30 sec - exiting 16:59:20 (2060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:04:44 (4656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:42:38 (4512): No heartbeat from core client for 30 sec - exiting 08:42:44 (4512): No heartbeat from core client for 30 sec - exiting 08:42:45 (4512): No heartbeat from core client for 30 sec - exiting 08:42:46 (4512): No heartbeat from core client for 30 sec - exiting 08:42:47 (4512): No heartbeat from core client for 30 sec - exiting 08:42:48 (4512): No heartbeat from core client for 30 sec - exiting 08:44:44 (4512): No heartbeat from core client for 30 sec - exiting 08:44:45 (4512): No heartbeat from core client for 30 sec - exiting 08:44:46 (4512): No heartbeat from core client for 30 sec - exiting 08:44:47 (4512): No heartbeat from core client for 30 sec - exiting 08:44:48 (4512): No heartbeat from core client for 30 sec - exiting 08:44:49 (4512): No heartbeat from core client for 30 sec - exiting 08:44:50 (4512): No heartbeat from core client for 30 sec - exiting 08:44:51 (4512): No heartbeat from core client for 30 sec - exiting 08:44:52 (4512): No heartbeat from core client for 30 sec - exiting 08:44:53 (4512): No heartbeat from core client for 30 sec - exiting 08:44:56 (4512): No heartbeat from core client for 30 sec - exiting 08:44:57 (4512): No heartbeat from core client for 30 sec - exiting 08:44:58 (4512): No heartbeat from core client for 30 sec - exiting 08:44:59 (4512): No heartbeat from core client for 30 sec - exiting 08:45:00 (4512): No heartbeat from core client for 30 sec - exiting 08:45:01 (4512): No heartbeat from core client for 30 sec - exiting 08:45:02 (4512): No heartbeat from core client for 30 sec - exiting 08:45:03 (4512): No heartbeat from core client for 30 sec - exiting 08:45:04 (4512): No heartbeat from core client for 30 sec - exiting 08:45:05 (4512): No heartbeat from core client for 30 sec - exiting 08:46:38 (4512): No heartbeat from core client for 30 sec - exiting 08:47:10 (4512): No heartbeat from core client for 30 sec - exiting 08:47:11 (4512): No heartbeat from core client for 30 sec - exiting 08:47:12 (4512): No heartbeat from core client for 30 sec - exiting 08:47:13 (4512): No heartbeat from core client for 30 sec - exiting 08:47:14 (4512): No heartbeat from core client for 30 sec - exiting 08:47:15 (4512): No heartbeat from core client for 30 sec - exiting 08:47:23 (4512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... 19:43:35 (4948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:56:37 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:56:38 (4664): No heartbeat from core client for 30 sec - exiting 00:57:39 (3992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:58:27 (4992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:40 (1020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:29 (4492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:02:05 (4428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:02:40 (4172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:03:40 (4408): No heartbeat from core client for 30 sec - exiting 01:03:41 (4408): No heartbeat from core client for 30 sec - exiting 01:03:42 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:04:37 (3140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:53:45 (2224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:54:29 (1412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:52:43 (3512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:53:19 (3376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:54:17 (2236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:55:21 (3204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:58:13 (4936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:01:27 (1536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_ugrz_1799_200_006654586/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_ugrz_1799_200_006654586/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_ugrz_1799_200_006654586/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_ugrz_1799_200_006654586/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_ugrz_1799_200_006654586/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_ugrz_1799_200_006654586/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy Sorry, too many model crashes! :-( 17:04:36 (4596): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Aug 2010 15:58:33	1057648	11671206	famous_ugrz_1799_200_006654586_5	702,026	211,161	0.3008
26 Aug 2010 15:53:24	1057648	11671206	famous_ugrz_1799_200_006654586_5	692,666	208,371	0.3008
26 Aug 2010 15:53:24	1057648	11671206	famous_ugrz_1799_200_006654586_5	683,306	205,589	0.3009
26 Aug 2010 13:33:00	1057648	11671206	famous_ugrz_1799_200_006654586_5	673,946	202,778	0.3009
26 Aug 2010 12:47:50	1057648	11671206	famous_ugrz_1799_200_006654586_5	664,586	199,990	0.3009
26 Aug 2010 11:54:47	1057648	11671206	famous_ugrz_1799_200_006654586_5	655,226	197,113	0.3008
26 Aug 2010 11:12:32	1057648	11671206	famous_ugrz_1799_200_006654586_5	645,866	194,352	0.3009
26 Aug 2010 10:25:13	1057648	11671206	famous_ugrz_1799_200_006654586_5	636,506	191,548	0.3009
26 Aug 2010 09:37:47	1057648	11671206	famous_ugrz_1799_200_006654586_5	627,146	188,766	0.3010
26 Aug 2010 09:00:44	1057648	11671206	famous_ugrz_1799_200_006654586_5	617,786	186,603	0.3021
26 Aug 2010 08:01:56	1057648	11671206	famous_ugrz_1799_200_006654586_5	608,426	183,180	0.3011
26 Aug 2010 07:13:48	1057648	11671206	famous_ugrz_1799_200_006654586_5	599,066	180,374	0.3011
26 Aug 2010 06:30:13	1057648	11671206	famous_ugrz_1799_200_006654586_5	589,706	177,583	0.3011
26 Aug 2010 05:36:47	1057648	11671206	famous_ugrz_1799_200_006654586_5	580,346	174,785	0.3012
26 Aug 2010 04:48:17	1057648	11671206	famous_ugrz_1799_200_006654586_5	570,986	171,989	0.3012
26 Aug 2010 04:04:31	1057648	11671206	famous_ugrz_1799_200_006654586_5	561,626	169,198	0.3013
26 Aug 2010 03:14:51	1057648	11671206	famous_ugrz_1799_200_006654586_5	552,266	166,403	0.3013
26 Aug 2010 02:27:33	1057648	11671206	famous_ugrz_1799_200_006654586_5	542,906	163,725	0.3016
26 Aug 2010 01:40:49	1057648	11671206	famous_ugrz_1799_200_006654586_5	533,546	160,946	0.3017
26 Aug 2010 01:27:51	1057648	11671206	famous_ugrz_1799_200_006654586_5	524,186	158,011	0.3014