Task 16662736

Name	hadcm3n_889q_1980_40_008720137_1
Workunit	8866115
Created	17 Jun 2014, 0:02:14 UTC
Sent	17 Jun 2014, 0:02:43 UTC
Report deadline	16 Sep 2014, 7:29:54 UTC
Received	1 Jul 2014, 20:56:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	976228
Run time	12 days 18 hours 36 min 43 sec
CPU time	11 days 23 hours 20 min 36 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.83 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.6.29</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:01:04 (67663): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:01:05 (67663): No heartbeat from core client for 30 sec - exiting 01:17:51 (26673): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:34:54 (26793): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:57:08 (26926): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(27106,0xa0029540) malloc: * error for object 0x2042204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27106,0xa0029540) malloc: * error for object 0x301f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27106,0xa0029540) malloc: * error for object 0x3831004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27106,0xa0029540) malloc: * error for object 0x701f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27106,0xa0029540) malloc: * error for object 0x781b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27106,0xa0029540) malloc: * error for object 0x5033404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27106,0xa0029540) malloc: * error for object 0x7836604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27106,0xa0029540) malloc: * error for object 0x7836600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27106,0xa0029540) malloc: * error for object 0x1801600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Tue Jul 1 13:25:25 2014 Thread 0 Crashed: 0 libSystem.B.dylib 0x9072db0f _small_free_list_remove_ptr + 246 1 libSystem.B.dylib 0x9072a546 _szone_free_definite_size + 3323 2 libSystem.B.dylib 0x907295e8 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000b99a __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3292 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000791c __Z8mainLoopv + 410 7 hadcm3n_6.07_i686-apple-darwin 0x000087c7 __Z6workerv + 2989 8 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 9 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x90722c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x907aa429 _nanosleep + 345 2 libSystem.B.dylib 0x907aa2ca _usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c __Z11boinc_sleepd + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x90750259 __pthread_start + 345 6 libSystem.B.dylib 0x907500de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8ed58 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /mnt/mystuff/guy/BOINC/slots/10/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90722000 - 0x908c9fff /usr/lib/libSystem.B.dylib 0x94d21000 - 0x94d24fff /usr/lib/system/libmathCommon.A.dylib 0x9911f000 - 0x99189fff /usr/lib/libstdc++.6.dylib 0x99263000 - 0x99271fff /usr/lib/libz.1.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Jul 2014 20:58:42	976228	16662736	hadcm3n_889q_1980_40_008720137_1	518,400	1,034,429	1.9954
01 Jul 2014 03:30:06	976228	16662736	hadcm3n_889q_1980_40_008720137_1	492,480	983,299	1.9966
30 Jun 2014 11:21:23	976228	16662736	hadcm3n_889q_1980_40_008720137_1	466,560	931,504	1.9965
29 Jun 2014 18:13:00	976228	16662736	hadcm3n_889q_1980_40_008720137_1	440,640	879,955	1.9970
29 Jun 2014 01:44:36	976228	16662736	hadcm3n_889q_1980_40_008720137_1	414,720	828,084	1.9967
28 Jun 2014 08:49:57	976228	16662736	hadcm3n_889q_1980_40_008720137_1	388,800	775,996	1.9959
27 Jun 2014 17:25:38	976228	16662736	hadcm3n_889q_1980_40_008720137_1	362,880	724,147	1.9956
27 Jun 2014 00:14:27	976228	16662736	hadcm3n_889q_1980_40_008720137_1	336,960	672,630	1.9962
26 Jun 2014 08:03:50	976228	16662736	hadcm3n_889q_1980_40_008720137_1	311,040	620,607	1.9953
25 Jun 2014 16:06:23	976228	16662736	hadcm3n_889q_1980_40_008720137_1	285,120	568,679	1.9945
24 Jun 2014 22:54:48	976228	16662736	hadcm3n_889q_1980_40_008720137_1	259,200	516,738	1.9936
24 Jun 2014 06:22:01	976228	16662736	hadcm3n_889q_1980_40_008720137_1	233,280	465,038	1.9935
23 Jun 2014 14:08:20	976228	16662736	hadcm3n_889q_1980_40_008720137_1	207,360	413,627	1.9947
23 Jun 2014 14:08:20	976228	16662736	hadcm3n_889q_1980_40_008720137_1	181,440	362,212	1.9963
23 Jun 2014 14:08:20	976228	16662736	hadcm3n_889q_1980_40_008720137_1	155,520	310,729	1.9980
21 Jun 2014 12:31:09	976228	16662736	hadcm3n_889q_1980_40_008720137_1	129,600	258,950	1.9981
20 Jun 2014 21:13:44	976228	16662736	hadcm3n_889q_1980_40_008720137_1	103,680	207,070	1.9972
20 Jun 2014 05:30:51	976228	16662736	hadcm3n_889q_1980_40_008720137_1	77,760	155,278	1.9969
19 Jun 2014 13:22:25	976228	16662736	hadcm3n_889q_1980_40_008720137_1	51,840	103,519	1.9969
18 Jun 2014 21:59:23	976228	16662736	hadcm3n_889q_1980_40_008720137_1	25,920	51,792	1.9981