Task 12976026

Name	hadcm3n_p5je_1900_40_007224762_2
Workunit	7423002
Created	14 Jun 2011, 18:24:16 UTC
Sent	14 Jun 2011, 18:24:21 UTC
Report deadline	14 Sep 2011, 1:51:32 UTC
Received	3 Aug 2011, 11:57:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1086188
Run time	5 days 21 hours 52 min 52 sec
CPU time	5 days 12 hours 54 min 27 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.10 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/p5jeko.pja5c10 is not a valid UM file. Error converting file to netcdf: dataout/p5jeko.pja5c10 Error: Input file: dataout/p5jeko.pia5c10 is not a valid UM file. Error converting file to netcdf: dataout/p5jeko.pia5c10 Error: Input file: dataout/p5jeko.pfa5c10 is not a valid UM file. Error converting file to netcdf: dataout/p5jeko.pfa5c10 Error: Input file: dataout/p5jeka.pha5c10 is not a valid UM file. Error converting file to netcdf: dataout/p5jeka.pha5c10 Error: Input file: dataout/p5jeka.pga5c10 is not a valid UM file. Error converting file to netcdf: dataout/p5jeka.pga5c10 Error: Input file: dataout/p5jeka.pea5c10 is not a valid UM file. Error converting file to netcdf: dataout/p5jeka.pea5c10 Error: Input file: dataout/p5jeka.pda5c10 is not a valid UM file. Error converting file to netcdf: dataout/p5jeka.pda5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(306,0xa00e2540) malloc: * error for object 0x82e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(306,0xa00e2540) malloc: * error for object 0x82e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(306,0xa00e2540) malloc: * error for object 0x1002404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(306,0xa00e2540) malloc: * error for object 0x1002400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(306,0xa00e2540) malloc: * error for object 0x1002404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(306,0xa00e2540) malloc: * error for object 0x1002400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(306,0xa00e2540) malloc: * error for object 0x1002404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(306,0xa00e2540) malloc: * error for object 0x1002400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(306,0xa00e2540) malloc: * error for object 0x1002404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(306,0xa00e2540) malloc: * error for object 0x1002400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(306,0xa00e2540) malloc: * error for object 0x1002404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K540 Fri Jul 22 07:59:07 2011 Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=316, selfPID=316, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x1816604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x1816600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x1001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x1001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x1001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x1001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x1001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x1001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x1001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x1001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x2800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(476,0xa0087540) malloc: * error for object 0x2800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_p5je_1900_40_007224762/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003E1BF7 Unknown Unknown Unknown hadcm3n_um_6.07_i 003DF4CC Unknown Unknown Unknown hadcm3n_um_6.07_i 0020E6E6 Unknown Unknown Unknown hadcm3n_um_6.07_i 00160E7A Unknown Unknown Unknown hadcm3n_um_6.07_i 00164453 Unknown Unknown Unknown hadcm3n_um_6.07_i 0038CE7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0038A1EDCalled boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 19:12:41	1086188	12976026	hadcm3n_p5je_1900_40_007224762_2	259,200	448,910	1.7319
25 Jul 2011 14:38:00	1086188	12976026	hadcm3n_p5je_1900_40_007224762_2	233,280	398,513	1.7083
09 Jul 2011 15:07:30	1086188	12976026	hadcm3n_p5je_1900_40_007224762_2	207,360	344,467	1.6612
08 Jul 2011 22:35:31	1086188	12976026	hadcm3n_p5je_1900_40_007224762_2	181,440	286,751	1.5804
08 Jul 2011 14:18:36	1086188	12976026	hadcm3n_p5je_1900_40_007224762_2	155,520	244,294	1.5708
24 Jun 2011 12:43:33	1086188	12976026	hadcm3n_p5je_1900_40_007224762_2	129,600	199,340	1.5381
21 Jun 2011 23:17:07	1086188	12976026	hadcm3n_p5je_1900_40_007224762_2	103,680	164,016	1.5819
21 Jun 2011 23:17:07	1086188	12976026	hadcm3n_p5je_1900_40_007224762_2	77,760	138,424	1.7801
19 Jun 2011 22:00:04	1086188	12976026	hadcm3n_p5je_1900_40_007224762_2	51,840	94,710	1.8270
16 Jun 2011 14:37:29	1086188	12976026	hadcm3n_p5je_1900_40_007224762_2	25,920	45,024	1.7370