Task 13193794

Name	hadcm3n_ybh6_1900_40_007347732_2
Workunit	7545162
Created	2 Aug 2011, 12:47:22 UTC
Sent	2 Aug 2011, 12:47:29 UTC
Report deadline	1 Nov 2011, 20:14:40 UTC
Received	27 Aug 2011, 9:53:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	850814
Run time	21 days 20 hours 35 min 56 sec
CPU time	12 days 10 hours 32 min 39 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	0.63 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:15:54 (7667): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:16:21 (7667): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:02:20 (25553): No heartbeat from core client for 30 sec - exiting 23:02:26 (25553): No heartbeat from core client for 30 sec - exiting 23:02:27 (25553): No heartbeat from core client for 30 sec - exiting 23:02:28 (25553): No heartbeat from core client for 30 sec - exiting 23:02:29 (25553): No heartbeat from core client for 30 sec - exiting 23:02:31 (25553): No heartbeat from core client for 30 sec - exiting 23:02:32 (25553): No heartbeat from core client for 30 sec - exiting 23:02:33 (25553): No heartbeat from core client for 30 sec - exiting 23:02:34 (25553): No heartbeat from core client for 30 sec - exiting 23:02:35 (25553): No heartbeat from core client for 30 sec - exiting 23:02:36 (25553): No heartbeat from core client for 30 sec - exiting 23:02:37 (25553): No heartbeat from core client for 30 sec - exiting 23:02:38 (25553): No heartbeat from core client for 30 sec - exiting 23:02:39 (25553): No heartbeat from core client for 30 sec - exiting 23:02:40 (25553): No heartbeat from core client for 30 sec - exiting 23:02:41 (25553): No heartbeat from core client for 30 sec - exiting 23:02:42 (25553): No heartbeat from core client for 30 sec - exiting 23:02:43 (25553): No heartbeat from core client for 30 sec - exiting 23:02:44 (25553): No heartbeat from core client for 30 sec - exiting 23:02:45 (25553): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/ybh6ko.pja2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ko.pja2c10 Error: Input file: dataout/ybh6ko.pia2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ko.pia2c10 Error: Input file: dataout/ybh6ko.pfa2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ko.pfa2c10 Error: Input file: dataout/ybh6ka.pha2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pha2c10 Error: Input file: dataout/ybh6ka.pga2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pga2c10 Error: Input file: dataout/ybh6ka.pea2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pea2c10 Error: Input file: dataout/ybh6ka.pda2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pda2c10 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/ybh6ko.pja2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ko.pja2c10 Error: Input file: dataout/ybh6ko.pia2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ko.pia2c10 Error: Input file: dataout/ybh6ko.pfa2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ko.pfa2c10 Error: Input file: dataout/ybh6ka.pha2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pha2c10 Error: Input file: dataout/ybh6ka.pga2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pga2c10 Error: Input file: dataout/ybh6ka.pea2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pea2c10 Error: Input file: dataout/ybh6ka.pda2c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pda2c10 02:30:09 (28598): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:42:48 (29232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:45:57 (30230): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:34:14 (32296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:56:50 (957): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:57:49 (3423): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:48:21 (5831): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:42:08 (6914): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/ybh6ko.pja3c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ko.pja3c10 Error: Input file: dataout/ybh6ko.pia3c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ko.pia3c10 Error: Input file: dataout/ybh6ko.pfa3c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ko.pfa3c10 Error: Input file: dataout/ybh6ka.pha3c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pha3c10 Error: Input file: dataout/ybh6ka.pga3c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pga3c10 Error: Input file: dataout/ybh6ka.pea3c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pea3c10 Error: Input file: dataout/ybh6ka.pda3c10 is not a valid UM file. Error converting file to netcdf: dataout/ybh6ka.pda3c10 14:05:09 (6974): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:05:16 (6974): No heartbeat from core client for 30 sec - exiting 17:30:41 (9094): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:20:00 (9586): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:43:11 (10133): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:02:27 (15840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:40:58 (16632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:05 (16849): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:21:44 (18267): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:02:09 (7084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:11 (7084): No heartbeat from core client for 30 sec - exiting 03:02:12 (7084): No heartbeat from core client for 30 sec - exiting 03:02:13 (7084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 10:10:57 (10605): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:11:02 (10605): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:53:22 (17539): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:37:00 (18553): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:54:41 (18785): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:55:11 (18785): No heartbeat from core client for 30 sec - exiting 05:08:13 (18837): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:37:19 (18875): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... cpdnmonitor: error reading file dataout/ocean_restart.day 05:41:09 (697): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file /home/wrweber/Desktop/BOINC/projects/climateprediction.net/hadcm3n_ybh6_1900_40_007347732/dataout/ocean_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file /home/wrweber/Desktop/BOINC/projects/climateprediction.net/hadcm3n_ybh6_1900_40_007347732/dataout/ocean_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file /home/wrweber/Desktop/BOINC/projects/climateprediction.net/hadcm3n_ybh6_1900_40_007347732/dataout/ocean_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file /home/wrweber/Desktop/BOINC/projects/climateprediction.net/hadcm3n_ybh6_1900_40_007347732/dataout/ocean_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file /home/wrweber/Desktop/BOINC/projects/climateprediction.net/hadcm3n_ybh6_1900_40_007347732/dataout/ocean_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file /home/wrweber/Desktop/BOINC/projects/climateprediction.net/hadcm3n_ybh6_1900_40_007347732/dataout/ocean_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Aug 2011 18:07:25	850814	13193794	hadcm3n_ybh6_1900_40_007347732_2	181,440	999,979	5.5113
22 Aug 2011 13:39:31	850814	13193794	hadcm3n_ybh6_1900_40_007347732_2	155,520	857,256	5.5122
18 Aug 2011 09:23:13	850814	13193794	hadcm3n_ybh6_1900_40_007347732_2	129,600	713,817	5.5078
15 Aug 2011 06:11:10	850814	13193794	hadcm3n_ybh6_1900_40_007347732_2	103,680	570,787	5.5053
12 Aug 2011 03:52:09	850814	13193794	hadcm3n_ybh6_1900_40_007347732_2	77,760	427,730	5.5006
09 Aug 2011 03:20:32	850814	13193794	hadcm3n_ybh6_1900_40_007347732_2	51,840	284,787	5.4936
05 Aug 2011 07:27:10	850814	13193794	hadcm3n_ybh6_1900_40_007347732_2	25,920	142,076	5.4813