Task 12832227

Name	hadcm3n_p6ly_1900_40_007226150_0
Workunit	7424390
Created	26 Apr 2011, 15:36:46 UTC
Sent	27 Apr 2011, 8:18:41 UTC
Report deadline	27 Jul 2011, 15:45:52 UTC
Received	14 May 2011, 21:56:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1143614
Run time	14 days 0 hours 3 min 46 sec
CPU time	11 days 2 hours 52 min 49 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.18 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:48:54 (34058): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:30:49 (2522): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:17:41 (39088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:53 (46529): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:06:22 (63277): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:39:25 (34488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:39:30 (34488): No heartbeat from core client for 30 sec - exiting 03:39:31 (34488): No heartbeat from core client for 30 sec - exiting 03:39:32 (34488): No heartbeat from core client for 30 sec - exiting 03:39:33 (34488): No heartbeat from core client for 30 sec - exiting 03:39:34 (34488): No heartbeat from core client for 30 sec - exiting 03:39:35 (34488): No heartbeat from core client for 30 sec - exiting 03:39:36 (34488): No heartbeat from core client for 30 sec - exiting 03:39:37 (34488): No heartbeat from core client for 30 sec - exiting 03:39:38 (34488): No heartbeat from core client for 30 sec - exiting 03:39:39 (34488): No heartbeat from core client for 30 sec - exiting 03:39:40 (34488): No heartbeat from core client for 30 sec - exiting 03:39:41 (34488): No heartbeat from core client for 30 sec - exiting 03:39:42 (34488): No heartbeat from core client for 30 sec - exiting 03:39:43 (34488): No heartbeat from core client for 30 sec - exiting 03:39:44 (34488): No heartbeat from core client for 30 sec - exiting 03:39:45 (34488): No heartbeat from core client for 30 sec - exiting 03:44:08 (43126): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:44:12 (43126): No heartbeat from core client for 30 sec - exiting 03:44:13 (43126): No heartbeat from core client for 30 sec - exiting 03:44:14 (43126): No heartbeat from core client for 30 sec - exiting 03:44:15 (43126): No heartbeat from core client for 30 sec - exiting 03:44:16 (43126): No heartbeat from core client for 30 sec - exiting 03:44:17 (43126): No heartbeat from core client for 30 sec - exiting 03:44:18 (43126): No heartbeat from core client for 30 sec - exiting 03:44:19 (43126): No heartbeat from core client for 30 sec - exiting 03:44:20 (43126): No heartbeat from core client for 30 sec - exiting 03:44:21 (43126): No heartbeat from core client for 30 sec - exiting 03:44:22 (43126): No heartbeat from core client for 30 sec - exiting 03:44:23 (43126): No heartbeat from core client for 30 sec - exiting 03:44:24 (43126): No heartbeat from core client for 30 sec - exiting 03:44:25 (43126): No heartbeat from core client for 30 sec - exiting 03:44:26 (43126): No heartbeat from core client for 30 sec - exiting 03:46:18 (43266): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:46:21 (43266): No heartbeat from core client for 30 sec - exiting 03:46:22 (43266): No heartbeat from core client for 30 sec - exiting 03:46:23 (43266): No heartbeat from core client for 30 sec - exiting 03:46:24 (43266): No heartbeat from core client for 30 sec - exiting 03:46:25 (43266): No heartbeat from core client for 30 sec - exiting 03:46:26 (43266): No heartbeat from core client for 30 sec - exiting 03:46:27 (43266): No heartbeat from core client for 30 sec - exiting 03:46:28 (43266): No heartbeat from core client for 30 sec - exiting 04:19:22 (43410): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:43:52 (52316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:43:58 (52316): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(80907,0xa0577540) malloc: * error for object 0xe005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80907,0xa0577540) malloc: * error for object 0xe005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80907,0xa0577540) malloc: * error for object 0x7009c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80907,0xa0577540) malloc: * error for object 0x7009c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80907,0xa0577540) malloc: * error for object 0x7009c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80907,0xa0577540) malloc: * error for object 0x7009c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80907,0xa0577540) malloc: * error for object 0x7009c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80907,0xa0577540) malloc: * error for object 0x7009c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80907,0xa0577540) malloc: * error for object 0x7009c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80907,0xa0577540) malloc: * error for object 0x7009c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80907,0xa0577540) malloc: * error for object 0xa004e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Sat May 14 17:55:49 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x93abf0af _small_free_list_remove_ptr + 246 1 libSystem.B.dylib 0x93abbb6c _szone_free_definite_size + 3457 2 libSystem.B.dylib 0x93abab88 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x00021e04 _ncio_px_get + 84 4 hadcm3n_6.07_i686-apple-darwin 0x000358a3 _fault_v1hs + 131 5 hadcm3n_6.07_i686-apple-darwin 0x0003644a _ncx_put_NC + 442 6 hadcm3n_6.07_i686-apple-darwin 0x0001ba64 _write_NC + 68 7 hadcm3n_6.07_i686-apple-darwin 0x0001bdbd _NC_endef + 269 8 hadcm3n_6.07_i686-apple-darwin 0x0001cdbb _nc_close + 75 9 hadcm3n_6.07_i686-apple-darwin 0x00013f95 __Z14add_attributesPKcS0_ + 1669 10 hadcm3n_6.07_i686-apple-darwin 0x0000bb26 __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3688 11 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 12 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 13 hadcm3n_6.07_i686-apple-darwin 0x0000791c __Z8mainLoopv + 410 14 hadcm3n_6.07_i686-apple-darwin 0x000087c7 __Z6workerv + 2989 15 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 16 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x93ab41ae _mach_wait_until + 10 1 libSystem.B.dylib 0x93b3b809 _nanosleep + 345 2 libSystem.B.dylib 0x93b3b6aa _usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c __Z11boinc_sleepd + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x93ae17fd __pthread_start + 345 6 libSystem.B.dylib 0x93ae1682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f258 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/9/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x93ab3000 - 0x93c5afff /usr/lib/libSystem.B.dylib 0x93ccb000 - 0x93cd9fff /usr/lib/libz.1.dylib 0x93f57000 - 0x93f5afff /usr/lib/system/libmathCommon.A.dylib 0x96b62000 - 0x96bccfff /usr/lib/libstdc++.6.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 May 2011 22:00:17	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	518,400	960,764	1.8533
14 May 2011 04:56:55	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	492,480	921,052	1.8702
13 May 2011 11:54:21	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	466,560	880,312	1.8868
12 May 2011 18:51:19	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	440,640	839,716	1.9057
11 May 2011 21:47:35	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	414,720	796,574	1.9208
10 May 2011 19:55:07	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	388,800	754,248	1.9399
10 May 2011 00:45:43	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	362,880	710,291	1.9574
09 May 2011 00:55:36	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	336,960	660,235	1.9594
08 May 2011 04:54:49	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	311,040	611,188	1.9650
06 May 2011 12:33:37	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	285,120	566,549	1.9871
05 May 2011 18:42:18	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	259,200	519,435	2.0040
04 May 2011 19:14:00	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	233,280	459,832	1.9712
03 May 2011 19:45:03	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	207,360	398,910	1.9238
02 May 2011 20:11:33	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	181,440	340,599	1.8772
01 May 2011 23:04:27	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	155,520	283,862	1.8252
01 May 2011 01:20:46	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	129,600	225,829	1.7425
30 Apr 2011 10:05:29	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	103,680	179,552	1.7318
29 Apr 2011 17:23:29	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	77,760	133,644	1.7187
29 Apr 2011 02:30:40	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	51,840	89,223	1.7211
28 Apr 2011 13:40:38	1143614	12832227	hadcm3n_p6ly_1900_40_007226150_0	25,920	45,040	1.7377