Task 15896313

Name	hadcm3n_493n_1940_40_008309908_3
Workunit	8461043
Created	18 Jul 2013, 16:53:12 UTC
Sent	18 Jul 2013, 17:02:48 UTC
Report deadline	18 Oct 2013, 0:29:59 UTC
Received	25 Aug 2013, 15:56:17 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1110299
Run time	18 days 10 hours 51 min 12 sec
CPU time	17 days 12 hours 24 min 49 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.92 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(493,0xac8372c0) malloc: * error for object 0x5343c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(493,0xac8372c0) malloc: * error for object 0x5343c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(493,0xac8372c0) malloc: * error for object 0x233ae04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(493,0xac8372c0) malloc: * error for object 0x233bc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(493,0xac8372c0) malloc: * error for object 0x233bc00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(493,0xac8372c0) malloc: * error for object 0x233bc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(493,0xac8372c0) malloc: * error for object 0x1332e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(493,0xac8372c0) malloc: * error for object 0x1332e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.5 build 11G63 Tue Aug 20 12:13:14 2013 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 hadcm3n_6.07_i686-apple-darwin(56017,0xac8372c0) malloc: * error for object 0x62c7604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(56017,0xac8372c0) malloc: * error for object 0x62c7600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Aug 2013 12:03:54	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	777,600	1,479,623	1.9028
19 Aug 2013 23:19:22	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	751,680	1,435,191	1.9093
19 Aug 2013 01:33:20	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	725,760	1,390,163	1.9155
18 Aug 2013 11:10:23	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	699,840	1,343,689	1.9200
17 Aug 2013 17:26:24	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	673,920	1,297,752	1.9257
16 Aug 2013 20:01:57	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	648,000	1,249,538	1.9283
15 Aug 2013 21:21:40	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	622,080	1,202,315	1.9327
15 Aug 2013 07:15:07	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	596,160	1,154,597	1.9367
14 Aug 2013 19:03:39	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	570,240	1,107,446	1.9421
14 Aug 2013 19:03:39	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	544,320	1,059,863	1.9471
14 Aug 2013 19:03:38	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	518,400	1,013,049	1.9542
14 Aug 2013 19:03:38	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	492,480	966,619	1.9628
14 Aug 2013 19:03:38	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	466,560	919,584	1.9710
14 Aug 2013 19:03:38	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	440,640	872,148	1.9793
14 Aug 2013 19:03:38	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	414,720	824,788	1.9888
14 Aug 2013 19:03:38	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	388,800	776,794	1.9979
14 Aug 2013 19:03:38	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	362,880	729,627	2.0107
14 Aug 2013 19:03:38	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	336,960	682,298	2.0249
14 Aug 2013 19:03:38	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	311,040	635,441	2.0430
14 Aug 2013 19:03:38	1110299	15896313	hadcm3n_493n_1940_40_008309908_3	285,120	588,376	2.0636