Task 13143723

Name	hadcm3n_ycw6_1900_40_007349568_2
Workunit	7546998
Created	17 Jul 2011, 15:27:05 UTC
Sent	17 Jul 2011, 15:35:50 UTC
Report deadline	16 Oct 2011, 23:03:01 UTC
Received	1 Aug 2011, 14:11:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1025983
Run time	6 days 9 hours 19 min 15 sec
CPU time	5 days 4 hours 42 min 26 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.88 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:27:59 (50803): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:28:00 (50803): No heartbeat from core client for 30 sec - exiting 22:28:01 (50803): No heartbeat from core client for 30 sec - exiting 22:28:02 (50803): No heartbeat from core client for 30 sec - exiting 22:28:03 (50803): No heartbeat from core client for 30 sec - exiting 22:28:04 (50803): No heartbeat from core client for 30 sec - exiting 22:28:05 (50803): No heartbeat from core client for 30 sec - exiting 22:28:06 (50803): No heartbeat from core client for 30 sec - exiting 22:28:07 (50803): No heartbeat from core client for 30 sec - exiting 22:28:08 (50803): No heartbeat from core client for 30 sec - exiting 22:28:09 (50803): No heartbeat from core client for 30 sec - exiting 22:28:10 (50803): No heartbeat from core client for 30 sec - exiting 22:28:11 (50803): No heartbeat from core client for 30 sec - exiting 22:28:12 (50803): No heartbeat from core client for 30 sec - exiting 22:28:13 (50803): No heartbeat from core client for 30 sec - exiting 22:28:14 (50803): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:50:02 (32241): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:14:43 (42919): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:08:56 (5374): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:08:57 (5374): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:52:32 (73290): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:52:34 (73290): No heartbeat from core client for 30 sec - exiting 17:52:35 (73290): No heartbeat from core client for 30 sec - exiting 17:52:36 (73290): No heartbeat from core client for 30 sec - exiting 17:52:37 (73290): No heartbeat from core client for 30 sec - exiting 17:52:38 (73290): No heartbeat from core client for 30 sec - exiting 17:52:39 (73290): No heartbeat from core client for 30 sec - exiting 17:52:40 (73290): No heartbeat from core client for 30 sec - exiting 17:52:41 (73290): No heartbeat from core client for 30 sec - exiting 17:52:42 (73290): No heartbeat from core client for 30 sec - exiting 17:52:43 (73290): No heartbeat from core client for 30 sec - exiting 17:52:44 (73290): No heartbeat from core client for 30 sec - exiting 17:52:45 (73290): No heartbeat from core client for 30 sec - exiting 17:52:46 (73290): No heartbeat from core client for 30 sec - exiting 17:52:47 (73290): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:55:46 (86719): No heartbeat from core client for 30 sec - exiting 12:55:47 (86719): No heartbeat from core client for 30 sec - exiting 12:55:48 (86719): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:01:04 (87131): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:52:11 (91009): No heartbeat from core client for 30 sec - exiting 13:52:12 (91009): No heartbeat from core client for 30 sec - exiting 13:52:13 (91009): No heartbeat from core client for 30 sec - exiting 13:52:14 (91009): No heartbeat from core client for 30 sec - exiting 13:52:15 (91009): No heartbeat from core client for 30 sec - exiting 13:52:16 (91009): No heartbeat from core client for 30 sec - exiting 13:52:17 (91009): No heartbeat from core client for 30 sec - exiting 13:52:18 (91009): No heartbeat from core client for 30 sec - exiting 13:52:19 (91009): No heartbeat from core client for 30 sec - exiting 13:52:20 (91009): No heartbeat from core client for 30 sec - exiting 13:52:21 (91009): No heartbeat from core client for 30 sec - exiting 13:52:22 (91009): No heartbeat from core client for 30 sec - exiting 13:52:23 (91009): No heartbeat from core client for 30 sec - exiting 13:52:24 (91009): No heartbeat from core client for 30 sec - exiting 13:52:25 (91009): No heartbeat from core client for 30 sec - exiting 13:52:26 (91009): No heartbeat from core client for 30 sec - exiting 13:52:27 (91009): No heartbeat from core client for 30 sec - exiting 13:52:28 (91009): No heartbeat from core client for 30 sec - exiting 13:52:29 (91009): No heartbeat from core client for 30 sec - exiting 13:52:30 (91009): No heartbeat from core client for 30 sec - exiting 13:52:31 (91009): No heartbeat from core client for 30 sec - exiting 13:52:32 (91009): No heartbeat from core client for 30 sec - exiting 13:52:33 (91009): No heartbeat from core client for 30 sec - exiting 13:52:34 (91009): No heartbeat from core client for 30 sec - exiting 13:52:35 (91009): No heartbeat from core client for 30 sec - exiting 13:52:36 (91009): No heartbeat from core client for 30 sec - exiting 13:52:37 (91009): No heartbeat from core client for 30 sec - exiting 13:52:38 (91009): No heartbeat from core client for 30 sec - exiting 13:52:39 (91009): No heartbeat from core client for 30 sec - exiting 13:52:40 (91009): No heartbeat from core client for 30 sec - exiting 13:52:41 (91009): No heartbeat from core client for 30 sec - exiting 13:52:42 (91009): No heartbeat from core client for 30 sec - exiting 13:52:43 (91009): No heartbeat from core client for 30 sec - exiting 13:52:44 (91009): No heartbeat from core client for 30 sec - exiting 13:52:45 (91009): No heartbeat from core client for 30 sec - exiting 13:52:46 (91009): No heartbeat from core client for 30 sec - exiting 13:52:47 (91009): No heartbeat from core client for 30 sec - exiting 13:52:48 (91009): No heartbeat from core client for 30 sec - exiting 13:52:49 (91009): No heartbeat from core client for 30 sec - exiting 13:52:50 (91009): No heartbeat from core client for 30 sec - exiting 13:52:51 (91009): No heartbeat from core client for 30 sec - exiting 13:52:52 (91009): No heartbeat from core client for 30 sec - exiting 13:52:53 (91009): No heartbeat from core client for 30 sec - exiting 13:52:54 (91009): No heartbeat from core client for 30 sec - exiting 13:52:55 (91009): No heartbeat from core client for 30 sec - exiting 13:52:56 (91009): No heartbeat from core client for 30 sec - exiting 13:52:57 (91009): No heartbeat from core client for 30 sec - exiting 13:52:58 (91009): No heartbeat from core client for 30 sec - exiting 13:52:59 (91009): No heartbeat from core client for 30 sec - exiting 13:53:00 (91009): No heartbeat from core client for 30 sec - exiting 13:53:01 (91009): No heartbeat from core client for 30 sec - exiting 13:53:02 (91009): No heartbeat from core client for 30 sec - exiting 13:53:03 (91009): No heartbeat from core client for 30 sec - exiting 13:53:04 (91009): No heartbeat from core client for 30 sec - exiting 13:53:05 (91009): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:28 (91235): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:57:57 (91291): No heartbeat from core client for 30 sec - exiting 13:57:58 (91291): No heartbeat from core client for 30 sec - exiting 13:57:59 (91291): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:03:36 (91381): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:05:41 (91756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(9670,0xa0a74540) malloc: * error for object 0x81b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(9670,0xa0a74540) malloc: * error for object 0x836c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(9670,0xa0a74540) malloc: * error for object 0x836c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(9670,0xa0a74540) malloc: * error for object 0x103c404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(9670,0xa0a74540) malloc: * error for object 0x81b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(9670,0xa0a74540) malloc: * error for object 0x836c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(9670,0xa0a74540) malloc: * error for object 0x836c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(9670,0xa0a74540) malloc: * error for object 0x3024a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9673, selfPID=9673, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(18391,0xa0a74540) malloc: * error for object 0x2805204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(18391,0xa0a74540) malloc: * error for object 0x2806004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(18391,0xa0a74540) malloc: * error for object 0x4018e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Aug 1 14:37:52 2011 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin. 0 libSystem.B.dylib 0x98958b03 small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x989555cc szone_free_definite_size + 3457 2 libSystem.B.dylib 0x989545e8 free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000876a worker() + 2896 7 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 8 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x9894dc0e mach_wait_until + 10 1 libSystem.B.dylib 0x989d5429 nanosleep + 345 2 libSystem.B.dylib 0x989d52ca usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c boinc_sleep(double) + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 timer_thread(void) + 78 5 libSystem.B.dylib 0x9897b259 _pthread_start + 345 6 libSystem.B.dylib 0x9897b0de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/1/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x91108000 - 0x9110bfff /usr/lib/system/libmathCommon.A.dylib 0x93694000 - 0x936fefff /usr/lib/libstdc++.6.dylib 0x9894d000 - 0x98af4fff /usr/lib/libSystem.B.dylib 0x9a79b000 - 0x9a7a9fff /usr/lib/libz.1.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Aug 2011 11:39:45	1025983	13143723	hadcm3n_ycw6_1900_40_007349568_2	259,200	446,734	1.7235
31 Jul 2011 17:47:57	1025983	13143723	hadcm3n_ycw6_1900_40_007349568_2	233,280	401,667	1.7218
30 Jul 2011 01:48:04	1025983	13143723	hadcm3n_ycw6_1900_40_007349568_2	207,360	357,149	1.7224
28 Jul 2011 17:50:17	1025983	13143723	hadcm3n_ycw6_1900_40_007349568_2	181,440	312,358	1.7215
27 Jul 2011 18:57:10	1025983	13143723	hadcm3n_ycw6_1900_40_007349568_2	155,520	267,720	1.7215
26 Jul 2011 21:39:39	1025983	13143723	hadcm3n_ycw6_1900_40_007349568_2	129,600	222,990	1.7206
25 Jul 2011 22:54:27	1025983	13143723	hadcm3n_ycw6_1900_40_007349568_2	103,680	178,263	1.7194
25 Jul 2011 22:16:29	1025983	13143723	hadcm3n_ycw6_1900_40_007349568_2	77,760	133,521	1.7171
25 Jul 2011 21:43:01	1025983	13143723	hadcm3n_ycw6_1900_40_007349568_2	51,840	88,853	1.7140
25 Jul 2011 19:25:29	1025983	13143723	hadcm3n_ycw6_1900_40_007349568_2	25,920	44,050	1.6995