Task 13901337

Name	hadcm3n_yhe6_1940_40_007547507_4
Workunit	7744739
Created	12 Jan 2012, 4:17:36 UTC
Sent	12 Jan 2012, 4:17:38 UTC
Report deadline	12 Apr 2012, 11:44:49 UTC
Received	26 Jan 2012, 13:14:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1152493
Run time	10 days 3 hours 11 min 20 sec
CPU time	2 days 17 hours 26 min 29 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	2.54 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:10:27 (2828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:10:29 (2828): No heartbeat from core client for 30 sec - exiting 19:19:41 (1128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:19:42 (1128): No heartbeat from core client for 30 sec - exiting 19:19:43 (1128): No heartbeat from core client for 30 sec - exiting 19:20:43 (5096): No heartbeat from core client for 30 sec - exiting 19:20:44 (5096): No heartbeat from core client for 30 sec - exiting 19:20:45 (5096): No heartbeat from core client for 30 sec - exiting 19:20:46 (5096): No heartbeat from core client for 30 sec - exiting 19:20:48 (5096): No heartbeat from core client for 30 sec - exiting 19:20:49 (5096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:21:29 (3008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:21:30 (3008): No heartbeat from core client for 30 sec - exiting 22:23:17 (1820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:23:18 (1820): No heartbeat from core client for 30 sec - exiting 23:24:19 (5952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:24:21 (5952): No heartbeat from core client for 30 sec - exiting 23:25:22 (5304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:25:23 (5304): No heartbeat from core client for 30 sec - exiting 23:25:24 (5304): No heartbeat from core client for 30 sec - exiting 23:25:25 (5304): No heartbeat from core client for 30 sec - exiting 01:39:14 (4932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:39:16 (4932): No heartbeat from core client for 30 sec - exiting 01:39:57 (756): No heartbeat from core client for 30 sec - exiting 01:39:58 (756): No heartbeat from core client for 30 sec - exiting 01:39:59 (756): No heartbeat from core client for 30 sec - exiting 01:40:00 (756): No heartbeat from core client for 30 sec - exiting 01:40:02 (756): No heartbeat from core client for 30 sec - exiting 01:40:03 (756): No heartbeat from core client for 30 sec - exiting 01:40:04 (756): No heartbeat from core client for 30 sec - exiting 01:40:05 (756): No heartbeat from core client for 30 sec - exiting 01:40:06 (756): No heartbeat from core client for 30 sec - exiting 01:40:07 (756): No heartbeat from core client for 30 sec - exiting 01:40:08 (756): No heartbeat from core client for 30 sec - exiting 01:40:09 (756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:40:10 (756): No heartbeat from core client for 30 sec - exiting 01:46:26 (1992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:48:58 (1992): No heartbeat from core client for 30 sec - exiting 01:49:07 (1992): No heartbeat from core client for 30 sec - exiting 01:49:11 (1992): No heartbeat from core client for 30 sec - exiting 01:49:21 (1992): No heartbeat from core client for 30 sec - exiting 01:51:50 (2808): No heartbeat from core client for 30 sec - exiting 01:51:51 (2808): No heartbeat from core client for 30 sec - exiting 01:51:52 (2808): No heartbeat from core client for 30 sec - exiting 01:51:53 (2808): No heartbeat from core client for 30 sec - exiting 01:51:54 (2808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:55:48 (3492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:55:53 (3492): No heartbeat from core client for 30 sec - exiting 01:55:54 (3492):01:58:34 (6080): No heartbeat from core client for 30 sec - exiting 01:58:35 (6080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:36 (6080): No heartbeat from core client for 30 sec - exiting 04:18:43 (5044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:18:45 (5044): No heartbeat from core client for 30 sec - exiting 05:01:11 (4108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:01:13 (4108): No heartbeat from core client for 30 sec - exiting 05:01:14 (4108): No heartbeat from core client for 30 sec - exiting 05:01:15 (4108): No heartbeat from core client for 30 sec - exiting 05:01:16 (4108): No heartbeat from core client for 30 sec - exiting 05:01:17 (4108): No heartbeat from core client for 30 sec - exiting 05:01:18 (4108): No heartbeat from core client for 30 sec - exiting 05:01:19 (4108): No heartbeat from core client for 30 sec - exiting 05:01:20 (4108): No heartbeat from core client for 30 sec - exiting 05:01:21 (4108): No heartbeat from core client for 30 sec - exiting 05:01:22 (4108): No heartbeat from core client for 30 sec - exiting 05:01:23 (4108): No heartbeat from core client for 30 sec - exiting 05:06:27 (5668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:06:41 (5668): No heartbeat from core client for 30 sec - exiting 05:06:46 (5668): No heartbeat from core client for 30 sec - exiting 05:06:49 (5668): No heartbeat from core client for 30 sec - exiting 05:06:50 (5668): No heartbeat from core client for 30 sec - exiting 05:06:53 (5668): No heartbeat from core client for 30 sec - exiting 05:06:55 (5668): No heartbeat from core client for 30 sec - exiting 05:07:01 (5668): No heartbeat from core client for 30 sec - exiting 05:10:51 (3292): No heartbeat from core client for 30 sec - exiting 05:10:52 (3292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:17:28 (5404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:17:51 (5404): No heartbeat from core client for 30 sec - exiting 05:18:13 (5404): No heartbeat from core client for 30 sec - exiting 05:18:14 (5404):05:58:37 (3728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:48 (4692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:49 (4692): No heartbeat from core client for 30 sec - exiting 06:38:48 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error ferror - Unit 67 - Return code = 32 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 no start tag in app init data 08:04:29 (4796): Can't parse init data file - running in standalone mode BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error ferror - Unit 21 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Write ERROR on history file for namelistNLIHISTO tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climatepredictioSignal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... no start tag in app init data 13:19:33 (5452): Can't parse init data file - running in standalone mode Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... forrtl: Not enough storage is available to process this command. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... forrtl: Not enough storage is available to process this command. CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5500, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Jan 2012 06:53:45	1152493	13901337	hadcm3n_yhe6_1940_40_007547507_4	181,440	275,342	1.5175
17 Jan 2012 18:28:55	1152493	13901337	hadcm3n_yhe6_1940_40_007547507_4	155,520	236,529	1.5209
16 Jan 2012 22:30:50	1152493	13901337	hadcm3n_yhe6_1940_40_007547507_4	129,600	197,665	1.5252
16 Jan 2012 09:59:56	1152493	13901337	hadcm3n_yhe6_1940_40_007547507_4	103,680	158,822	1.5318
15 Jan 2012 21:21:02	1152493	13901337	hadcm3n_yhe6_1940_40_007547507_4	77,760	119,037	1.5308
15 Jan 2012 09:40:39	1152493	13901337	hadcm3n_yhe6_1940_40_007547507_4	51,840	80,282	1.5486
13 Jan 2012 23:04:52	1152493	13901337	hadcm3n_yhe6_1940_40_007547507_4	25,920	40,264	1.5534