Task 16589216

Name	hadcm3n_8bji_1980_40_008724377_0
Workunit	8870355
Created	23 Apr 2014, 13:12:34 UTC
Sent	2 May 2014, 21:31:39 UTC
Report deadline	2 Aug 2014, 4:58:50 UTC
Received	9 Jun 2014, 6:56:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1055843
Run time	24 days 9 hours 56 min 6 sec
CPU time	14 days 8 hours 11 min 7 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.81 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:53:12 (72664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:53:14 (72664): No heartbeat from core client for 30 sec - exiting 20:53:15 (72664): No heartbeat from core client for 30 sec - exiting 20:53:16 (72664): No heartbeat from core client for 30 sec - exiting 20:53:17 (72664): No heartbeat from core client for 30 sec - exiting 20:53:18 (72664): No heartbeat from core client for 30 sec - exiting 20:53:19 (72664): No heartbeat from core client for 30 sec - exiting 20:53:20 (72664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:21:52 (40320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:58:21 (46958): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:57:08 (92384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(76495,0xa04901a8) malloc: * error for object 0x915e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(76495,0xa04901a8) malloc: * error for object 0x915e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(88723,0xa04901a8) malloc: * error for object 0x96d004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(88723,0xa04901a8) malloc: * error for object 0x93f804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0xc3fa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0x1416c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0x13f9004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0x13f9000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0xbeb804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0xbeb800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0xbeb804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0xbeb800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0xbeb804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0xbeb800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0xbeb804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0xbeb800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0xbeb804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(42727,0xa03181a8) malloc: * error for object 0xbeb800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... 21:09:41 (51226): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:02:10 (41481): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:34:21 (24080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:38:47 (35285): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:45:48 (84132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:40:12 (13220): No heartbeat from core client for 30 sec - exiting 14:40:13 (13220): No heartbeat from core client for 30 sec - exiting 14:40:14 (13220): No heartbeat from core client for 30 sec - exiting 14:40:15 (13220): No heartbeat from core client for 30 sec - exiting 14:40:17 (13220): No heartbeat from core client for 30 sec - exiting 14:40:18 (13220): No heartbeat from core client for 30 sec - exiting 14:40:19 (13220): No heartbeat from core client for 30 sec - exiting 14:40:20 (13220): No heartbeat from core client for 30 sec - exiting 14:40:21 (13220): No heartbeat from core client for 30 sec - exiting 14:40:22 (13220): No heartbeat from core client for 30 sec - exiting 14:40:23 (13220): No heartbeat from core client for 30 sec - exiting 14:40:24 (13220): No heartbeat from core client for 30 sec - exiting 14:40:25 (13220): No heartbeat from core client for 30 sec - exiting 14:40:26 (13220): No heartbeat from core client for 30 sec - exiting 14:40:27 (13220): No heartbeat from core client for 30 sec - exiting 14:40:28 (13220): No heartbeat from core client for 30 sec - exiting 14:40:29 (13220): No heartbeat from core client for 30 sec - exiting 14:40:30 (13220): No heartbeat from core client for 30 sec - exiting 14:40:31 (13220): No heartbeat from core client for 30 sec - exiting 14:40:32 (13220): No heartbeat from core client for 30 sec - exiting 14:40:33 (13220): No heartbeat from core client for 30 sec - exiting 14:40:34 (13220): No heartbeat from core client for 30 sec - exiting 14:40:35 (13220): No heartbeat from core client for 30 sec - exiting 14:40:36 (13220): No heartbeat from core client for 30 sec - exiting 14:40:37 (13220): No heartbeat from core client for 30 sec - exiting 14:40:38 (13220): No heartbeat from core client for 30 sec - exiting 14:40:39 (13220): No heartbeat from core client for 30 sec - exiting 14:40:40 (13220): No heartbeat from core client for 30 sec - exiting 14:40:41 (13220): No heartbeat from core client for 30 sec - exiting 14:40:42 (13220): No heartbeat from core client for 30 sec - exiting 14:40:43 (13220): No heartbeat from core client for 30 sec - exiting 14:40:44 (13220): No heartbeat from core client for 30 sec - exiting 14:40:45 (13220): No heartbeat from core client for 30 sec - exiting 14:40:46 (13220): No heartbeat from core client for 30 sec - exiting 14:40:47 (13220): No heartbeat from core client for 30 sec - exiting 14:40:48 (13220): No heartbeat from core client for 30 sec - exiting 14:40:49 (13220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_8bji_1980_40_008724377/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Jun 2014 09:01:57	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	777,600	1,239,096	1.5935
09 Jun 2014 06:53:34	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	751,680	1,197,367	1.5929
05 Jun 2014 12:27:13	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	725,760	1,155,523	1.5922
03 Jun 2014 19:33:38	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	699,840	1,113,789	1.5915
02 Jun 2014 12:38:03	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	673,920	1,071,633	1.5901
01 Jun 2014 15:51:29	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	648,000	1,029,813	1.5892
31 May 2014 03:25:22	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	622,080	989,361	1.5904
29 May 2014 18:39:00	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	596,160	948,481	1.5910
28 May 2014 11:18:07	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	570,240	907,905	1.5921
27 May 2014 15:51:25	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	544,320	867,249	1.5933
26 May 2014 15:57:30	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	518,400	827,692	1.5966
25 May 2014 18:44:56	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	492,480	787,024	1.5981
24 May 2014 20:14:01	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	466,560	746,674	1.6004
23 May 2014 19:58:40	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	440,640	707,772	1.6062
23 May 2014 04:54:40	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	414,720	667,598	1.6098
21 May 2014 19:04:31	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	388,800	626,983	1.6126
20 May 2014 02:15:20	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	362,880	585,922	1.6146
19 May 2014 00:26:24	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	336,960	544,689	1.6165
17 May 2014 19:10:51	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	311,040	502,781	1.6165
16 May 2014 19:14:08	1055843	16589216	hadcm3n_8bji_1980_40_008724377_0	285,120	461,338	1.6180