Task 13097616

Name	hadcm3n_yatd_1900_40_007346875_0
Workunit	7544305
Created	6 Jul 2011, 13:40:34 UTC
Sent	19 Jul 2011, 9:58:25 UTC
Report deadline	18 Oct 2011, 17:25:36 UTC
Received	15 Aug 2011, 6:28:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	942621
Run time	6 days 1 hours 56 min 20 sec
CPU time	4 days 17 hours 6 min 52 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.42 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.6.36</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:46:54 (7554): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:46:55 (7554): No heartbeat from core client for 30 sec - exiting 02:46:56 (7554): No heartbeat from core client for 30 sec - exiting 02:46:57 (7554): No heartbeat from core client for 30 sec - exiting 02:46:58 (7554): No heartbeat from core client for 30 sec - exiting 02:46:59 (7554): No heartbeat from core client for 30 sec - exiting 02:47:00 (7554): No heartbeat from core client for 30 sec - exiting 02:47:01 (7554): No heartbeat from core client for 30 sec - exiting 02:47:02 (7554): No heartbeat from core client for 30 sec - exiting 02:47:03 (7554): No heartbeat from core client for 30 sec - exiting 02:47:04 (7554): No heartbeat from core client for 30 sec - exiting 02:47:05 (7554): No heartbeat from core client for 30 sec - exiting 02:47:06 (7554): No heartbeat from core client for 30 sec - exiting 02:47:07 (7554): No heartbeat from core client for 30 sec - exiting 02:47:08 (7554): No heartbeat from core client for 30 sec - exiting 02:47:09 (7554): No heartbeat from core client for 30 sec - exiting 02:47:10 (7554): No heartbeat from core client for 30 sec - exiting 02:47:11 (7554): No heartbeat from core client for 30 sec - exiting 02:47:12 (7554): No heartbeat from core client for 30 sec - exiting 02:47:13 (7554): No heartbeat from core client for 30 sec - exiting 02:47:14 (7554): No heartbeat from core client for 30 sec - exiting 02:47:15 (7554): No heartbeat from core client for 30 sec - exiting 02:47:16 (7554): No heartbeat from core client for 30 sec - exiting 02:47:17 (7554): No heartbeat from core client for 30 sec - exiting 02:47:18 (7554): No heartbeat from core client for 30 sec - exiting 02:47:19 (7554): No heartbeat from core client for 30 sec - exiting 02:47:20 (7554): No heartbeat from core client for 30 sec - exiting 02:47:21 (7554): No heartbeat from core client for 30 sec - exiting 02:47:22 (7554): No heartbeat from core client for 30 sec - exiting 02:47:23 (7554): No heartbeat from core client for 30 sec - exiting 02:47:24 (7554): No heartbeat from core client for 30 sec - exiting 02:47:25 (7554): No heartbeat from core client for 30 sec - exiting 02:47:26 (7554): No heartbeat from core client for 30 sec - exiting 02:47:27 (7554): No heartbeat from core client for 30 sec - exiting 02:47:28 (7554): No heartbeat from core client for 30 sec - exiting 02:47:29 (7554): No heartbeat from core client for 30 sec - exiting 02:47:30 (7554): No heartbeat from core client for 30 sec - exiting 02:47:31 (7554): No heartbeat from core client for 30 sec - exiting 02:47:32 (7554): No heartbeat from core client for 30 sec - exiting 02:47:33 (7554): No heartbeat from core client for 30 sec - exiting 02:47:34 (7554): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x381f004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x381f000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x383a604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x383a600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x825a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x841004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x841000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x383a604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x383a600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x2039c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x383a604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x383a600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x1034c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x1034c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x283aa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x4002204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x4002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(32608,0xa009d540) malloc: * error for object 0x4002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Aug 15 06:50:14 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x909b5b0f _small_free_list_remove_ptr + 246 1 libSystem.B.dylib 0x909b25cc _szone_free_definite_size + 3457 2 libSystem.B.dylib 0x909b15e8 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000791c __Z8mainLoopv + 410 7 hadcm3n_6.07_i686-apple-darwin 0x000087c7 __Z6workerv + 2989 8 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 9 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x909aac0e _mach_wait_until + 10 1 libSystem.B.dylib 0x90a32429 _nanosleep + 345 2 libSystem.B.dylib 0x90a322ca _usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c __Z11boinc_sleepd + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x909d8259 __pthread_start + 345 6 libSystem.B.dylib 0x909d80de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f7e8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/2/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x909aa000 - 0x90b51fff /usr/lib/libSystem.B.dylib 0x92adb000 - 0x92adefff /usr/lib/system/libmathCommon.A.dylib 0x92adf000 - 0x92b49fff /usr/lib/libstdc++.6.dylib 0x9a940000 - 0x9a94efff /usr/lib/libz.1.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Aug 2011 06:31:39	942621	13097616	hadcm3n_yatd_1900_40_007346875_0	259,200	407,206	1.5710
14 Aug 2011 13:12:51	942621	13097616	hadcm3n_yatd_1900_40_007346875_0	233,280	367,081	1.5736
13 Aug 2011 22:52:10	942621	13097616	hadcm3n_yatd_1900_40_007346875_0	207,360	326,963	1.5768
13 Aug 2011 07:22:29	942621	13097616	hadcm3n_yatd_1900_40_007346875_0	181,440	286,917	1.5813
12 Aug 2011 15:27:49	942621	13097616	hadcm3n_yatd_1900_40_007346875_0	155,520	246,365	1.5841
11 Aug 2011 14:24:41	942621	13097616	hadcm3n_yatd_1900_40_007346875_0	129,600	206,275	1.5916
10 Aug 2011 12:39:01	942621	13097616	hadcm3n_yatd_1900_40_007346875_0	103,680	164,684	1.5884
09 Aug 2011 22:16:56	942621	13097616	hadcm3n_yatd_1900_40_007346875_0	77,760	124,416	1.6000
09 Aug 2011 06:36:00	942621	13097616	hadcm3n_yatd_1900_40_007346875_0	51,840	86,371	1.6661
25 Jul 2011 18:48:50	942621	13097616	hadcm3n_yatd_1900_40_007346875_0	25,920	43,032	1.6602