Task 14292277

Name	hadcm3n_u5y2_1980_40_007834123_2
Workunit	7989235
Created	19 Mar 2012, 16:08:12 UTC
Sent	19 Mar 2012, 16:10:00 UTC
Report deadline	18 Jun 2012, 23:37:11 UTC
Received	29 Aug 2012, 4:34:41 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	812178
Run time	10 days 3 hours 50 min 5 sec
CPU time	7 days 19 hours 11 min 1 sec
Validate state	Invalid
Credit	4,354.56
Device peak FLOPS	2.12 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:52:21 (2056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3440, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:13:21 (2768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:14:57 (1732): No heartbeat from core client for 30 sec - exiting 21:14:58 (1732): No heartbeat from core client for 30 sec - exiting 21:14:59 (1732): No heartbeat from core client for 30 sec - exiting 21:15:00 (1732): No heartbeat from core client for 30 sec - exiting 21:15:01 (1732): No heartbeat from core client for 30 sec - exiting 21:15:02 (1732): No heartbeat from core client for 30 sec - exiting 21:15:03 (1732): No heartbeat from core client for 30 sec - exiting 21:15:04 (1732): No heartbeat from core client for 30 sec - exiting 21:15:05 (1732): No heartbeat from core client for 30 sec - exiting 21:15:06 (1732): No heartbeat from core client for 30 sec - exiting 21:15:07 (1732): No heartbeat from core client for 30 sec - exiting 21:15:08 (1732): No heartbeat from core client for 30 sec - exiting 21:15:09 (1732): No heartbeat from core client for 30 sec - exiting 21:15:10 (1732): No heartbeat from core client for 30 sec - exiting 21:15:11 (1732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/u5y2ko.pjj3c10 Error converting file to netcdf: dataout/u5y2ko.pij3c10 Error converting file to netcdf: dataout/u5y2ko.pfj3c10 Error converting file to netcdf: dataout/u5y2ka.phj3c10 Error converting file to netcdf: dataout/u5y2ka.pgj3c10 Error converting file to netcdf: dataout/u5y2ka.pej3c10 Error converting file to netcdf: dataout/u5y2ka.pdj3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3560, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Aug 2012 23:39:09	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	362,880	659,669	1.8179
14 Aug 2012 07:51:18	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	336,960	619,917	1.8397
11 Jul 2012 09:18:46	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	311,040	573,865	1.8450
30 Jun 2012 09:52:47	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	285,120	529,269	1.8563
16 May 2012 11:20:26	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	259,200	485,452	1.8729
13 May 2012 22:19:45	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	233,280	433,043	1.8563
10 May 2012 11:33:02	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	207,360	380,818	1.8365
08 May 2012 13:18:55	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	181,440	326,600	1.8000
03 May 2012 11:02:45	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	155,520	283,189	1.8209
27 Apr 2012 08:52:00	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	129,600	234,312	1.8080
27 Mar 2012 04:59:39	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	103,680	193,272	1.8641
23 Mar 2012 21:54:23	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	77,760	141,679	1.8220
23 Mar 2012 06:58:37	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	51,840	90,356	1.7430
22 Mar 2012 07:04:15	812178	14292277	hadcm3n_u5y2_1980_40_007834123_2	25,920	49,227	1.8992