Task 13395019

Name	hadcm3n_t187_1940_40_007445627_4
Workunit	7643130
Created	18 Sep 2011, 2:34:22 UTC
Sent	18 Sep 2011, 2:37:15 UTC
Report deadline	18 Dec 2011, 10:04:26 UTC
Received	15 Nov 2011, 18:38:14 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1165892
Run time	24 days 16 hours 49 min 28 sec
CPU time	20 days 18 hours 56 min 20 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	1.99 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/t187ko.pjg6c10 is not a valid UM file. Error converting file to netcdf: dataout/t187ko.pjg6c10 Error: Input file: dataout/t187ko.pig6c10 is not a valid UM file. Error converting file to netcdf: dataout/t187ko.pig6c10 Error: Input file: dataout/t187ko.pfg6c10 is not a valid UM file. Error converting file to netcdf: dataout/t187ko.pfg6c10 Error: Input file: dataout/t187ka.phg6c10 is not a valid UM file. Error converting file to netcdf: dataout/t187ka.phg6c10 Error: Input file: dataout/t187ka.pgg6c10 is not a valid UM file. Error converting file to netcdf: dataout/t187ka.pgg6c10 Error: Input file: dataout/t187ka.peg6c10 is not a valid UM file. Error converting file to netcdf: dataout/t187ka.peg6c10 Error: Input file: dataout/t187ka.pdg6c10 is not a valid UM file. Error converting file to netcdf: dataout/t187ka.pdg6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(55190,0xac4502c0) malloc: * error for object 0x129ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=55192, selfPID=55192, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(59645,0xac4502c0) malloc: * error for object 0x1314600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t187_1940_40_007445627/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Nov 2011 18:43:53	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	777,600	1,782,158	2.2919
09 Nov 2011 21:51:33	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	751,680	1,722,498	2.2915
08 Nov 2011 22:51:42	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	725,760	1,661,915	2.2899
07 Nov 2011 16:15:15	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	699,840	1,605,183	2.2936
05 Nov 2011 14:42:14	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	673,920	1,549,553	2.2993
04 Nov 2011 12:40:25	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	648,000	1,491,941	2.3024
03 Nov 2011 08:49:20	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	622,080	1,434,579	2.3061
02 Nov 2011 12:17:31	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	596,160	1,377,060	2.3099
01 Nov 2011 15:23:39	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	570,240	1,316,939	2.3094
31 Oct 2011 19:44:37	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	544,320	1,256,995	2.3093
31 Oct 2011 19:03:37	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	518,400	1,200,807	2.3164
31 Oct 2011 18:33:34	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	492,480	1,142,953	2.3208
31 Oct 2011 17:40:34	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	466,560	1,085,669	2.3270
31 Oct 2011 17:12:43	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	440,640	1,027,778	2.3325
31 Oct 2011 16:42:55	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	414,720	970,476	2.3401
31 Oct 2011 15:31:11	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	388,800	912,694	2.3475
31 Oct 2011 13:56:45	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	362,880	854,277	2.3542
31 Oct 2011 13:56:44	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	336,960	797,358	2.3663
31 Oct 2011 13:56:43	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	311,040	739,741	2.3783
31 Oct 2011 13:56:41	1165892	13395019	hadcm3n_t187_1940_40_007445627_4	285,120	682,048	2.3921