Task 12805315

Name	hadcm3n_o22i_1900_40_007198013_2
Workunit	7396293
Created	21 Apr 2011, 11:00:36 UTC
Sent	21 Apr 2011, 11:02:34 UTC
Report deadline	21 Jul 2011, 18:29:45 UTC
Received	21 Jun 2011, 0:48:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1087383
Run time	25 days 14 hours 14 min 49 sec
CPU time	19 days 8 hours 45 min 24 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	1.40 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:13:04 (1348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:30 (1348): No heartbeat from core client for 30 sec - exiting 09:13:35 (1348): No heartbeat from core client for 30 sec - exiting 09:13:36 (1348): No heartbeat from core client for 30 sec - exiting 09:23:54 (6116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:35:02 (3044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:48:01 (3700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:48:12 (3700): No heartbeat from core client for 30 sec - exiting 13:54:52 (3884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:56:41 (3916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:57:55 (3916): No heartbeat from core client for 30 sec - exiting 14:02:55 (5592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:03:52 (5592): No heartbeat from core client for 30 sec - exiting 14:24:48 (720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:24:55 (720): No heartbeat from core client for 30 sec - exiting 14:28:46 (3904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:29:50 (3904):CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN MoNo Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=436, selfPID=436, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5328, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5328, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5328, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5328, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5328, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5328, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Jun 2011 17:45:32	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	492,480	1,641,251	3.3326
17 Jun 2011 07:35:09	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	466,560	1,555,018	3.3329
15 Jun 2011 23:32:30	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	440,640	1,465,695	3.3263
14 Jun 2011 06:55:53	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	414,720	1,374,559	3.3144
11 Jun 2011 14:32:24	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	388,800	1,282,249	3.2980
26 May 2011 20:49:40	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	362,880	1,197,498	3.3000
24 May 2011 22:17:57	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	336,960	1,106,325	3.2833
23 May 2011 15:21:44	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	311,040	1,019,846	3.2788
14 May 2011 20:59:38	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	285,120	934,711	3.2783
13 May 2011 15:24:05	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	259,200	850,241	3.2803
12 May 2011 11:04:27	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	233,280	766,969	3.2878
11 May 2011 07:33:57	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	207,360	681,718	3.2876
09 May 2011 15:53:17	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	181,440	596,676	3.2886
07 May 2011 16:12:44	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	155,520	513,820	3.3039
05 May 2011 22:49:45	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	129,600	428,657	3.3075
04 May 2011 20:10:38	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	103,680	345,480	3.3322
28 Apr 2011 06:12:44	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	77,760	260,313	3.3476
26 Apr 2011 23:13:08	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	51,840	174,034	3.3571
25 Apr 2011 06:23:42	1087383	12805315	hadcm3n_o22i_1900_40_007198013_2	25,920	87,668	3.3823