Task 12758764

Name	hadcm3n_o42j_1900_40_007200606_2
Workunit	7398886
Created	31 Mar 2011, 8:26:34 UTC
Sent	31 Mar 2011, 8:37:31 UTC
Report deadline	30 Jun 2011, 16:04:42 UTC
Received	23 Apr 2011, 13:35:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	895482
Run time	12 days 0 hours 12 min 37 sec
CPU time	9 days 13 hours 53 min 36 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.24 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:16:48 (65194): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:16:49 (65194): No heartbeat from core client for 30 sec - exiting 01:16:50 (65194): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:29:41 (81297): No heartbeat from core client for 30 sec - exiting 13:29:43 (81297): No heartbeat from core client for 30 sec - exiting 13:29:44 (81297): No heartbeat from core client for 30 sec - exiting 13:29:45 (81297): No heartbeat from core client for 30 sec - exiting 13:29:46 (81297): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:57:53 (93479): No heartbeat from core client for 30 sec - exiting 17:57:54 (93479): No heartbeat from core client for 30 sec - exiting 17:57:55 (93479): No heartbeat from core client for 30 sec - exiting 17:57:56 (93479): No heartbeat from core client for 30 sec - exiting 17:57:57 (93479): No heartbeat from core client for 30 sec - exiting 17:57:58 (93479): No heartbeat from core client for 30 sec - exiting 17:57:59 (93479): No heartbeat from core client for 30 sec - exiting 17:58:00 (93479): No heartbeat from core client for 30 sec - exiting 17:58:01 (93479): No heartbeat from core client for 30 sec - exiting 17:58:02 (93479): No heartbeat from core client for 30 sec - exiting 17:58:03 (93479): No heartbeat from core client for 30 sec - exiting 17:58:04 (93479): No heartbeat from core client for 30 sec - exiting 17:58:05 (93479): No heartbeat from core client for 30 sec - exiting 17:58:06 (93479): No heartbeat from core client for 30 sec - exiting 17:58:07 (93479): No heartbeat from core client for 30 sec - exiting 17:58:08 (93479): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(84367,0xa0149540) malloc: * error for object 0x810e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84367,0xa0149540) malloc: * error for object 0x811c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84367,0xa0149540) malloc: * error for object 0x811c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84367,0xa0149540) malloc: * error for object 0x811c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=84375, selfPID=84375, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(93327,0xa0149540) malloc: * error for object 0x2031404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(93327,0xa0149540) malloc: * error for object 0x2031400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(93327,0xa0149540) malloc: * error for object 0x2031404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(93327,0xa0149540) malloc: * error for object 0x2031400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(93327,0xa0149540) malloc: * error for object 0x2805804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(93327,0xa0149540) malloc: * error for object 0x2806604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(93327,0xa0149540) malloc: * error for object 0x2806600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(93327,0xa0149540) malloc: * error for object 0x2806604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(93327,0xa0149540) malloc: * error for object 0x2806600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(93327,0xa0149540) malloc: * error for object 0x2806604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(93327,0xa0149540) malloc: * error for object 0x2806600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:08:57 (751): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:13:25 (10548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:31:40 (17521): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:40:27 (17702): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:42:27 (17809): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:42:28 (17809): No heartbeat from core client for 30 sec - exiting 00:42:29 (17809): No heartbeat from core client for 30 sec - exiting 00:42:30 (17809): No heartbeat from core client for 30 sec - exiting 00:42:31 (17809): No heartbeat from core client for 30 sec - exiting 00:42:32 (17809): No heartbeat from core client for 30 sec - exiting 00:42:33 (17809): No heartbeat from core client for 30 sec - exiting 00:45:18 (17827): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:45:20 (17827): No heartbeat from core client for 30 sec - exiting 00:45:21 (17827): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:24:44 (33753): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:27:43 (30743): No heartbeat from core client for 30 sec - exiting 10:27:44 (30743): No heartbeat from core client for 30 sec - exiting 10:27:45 (30743): No heartbeat from core client for 30 sec - exiting 10:27:46 (30743): No heartbeat from core client for 30 sec - exiting 10:27:47 (30743): No heartbeat from core client for 30 sec - exiting 10:27:48 (30743): No heartbeat from core client for 30 sec - exiting 10:27:49 (30743): No heartbeat from core client for 30 sec - exiting 10:27:50 (30743): No heartbeat from core client for 30 sec - exiting 10:27:51 (30743): No heartbeat from core client for 30 sec - exiting 10:27:52 (30743): No heartbeat from core client for 30 sec - exiting 10:27:53 (30743): No heartbeat from core client for 30 sec - exiting 10:27:54 (30743): No heartbeat from core client for 30 sec - exiting 10:27:55 (30743): No heartbeat from core client for 30 sec - exiting 10:27:56 (30743): No heartbeat from core client for 30 sec - exiting 10:27:57 (30743): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(34393,0xa0808540) malloc: * error for object 0x2012604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34393,0xa0808540) malloc: * error for object 0x2012600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34393,0xa0808540) malloc: * error for object 0x2802004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34393,0xa0808540) malloc: * error for object 0x2802000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(34393,0xa0808540) malloc: * error for object 0x2802004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Sat Apr 23 15:34:29 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x95c350a3 _small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x95c31b6c _szone_free_definite_size + 3457 2 libSystem.B.dylib 0x95c30b88 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000791c __Z8mainLoopv + 410 7 hadcm3n_6.07_i686-apple-darwin 0x000087c7 __Z6workerv + 2989 8 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 9 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x95c2a1ae _mach_wait_until + 10 1 libSystem.B.dylib 0x95cb1809 _nanosleep + 345 2 libSystem.B.dylib 0x95cb16aa _usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c __Z11boinc_sleepd + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x95c577fd __pthread_start + 345 6 libSystem.B.dylib 0x95c57682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f708 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/13/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x9062c000 - 0x9063afff /usr/lib/libz.1.dylib 0x95c29000 - 0x95dd0fff /usr/lib/libSystem.B.dylib 0x96869000 - 0x9686cfff /usr/lib/system/libmathCommon.A.dylib 0x9722a000 - 0x97294fff /usr/lib/libstdc++.6.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Apr 2011 13:36:46	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	518,400	827,607	1.5965
23 Apr 2011 02:38:41	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	492,480	792,581	1.6094
22 Apr 2011 15:09:58	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	466,560	757,532	1.6237
21 Apr 2011 22:20:53	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	440,640	721,423	1.6372
21 Apr 2011 03:15:44	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	414,720	684,010	1.6493
20 Apr 2011 15:49:25	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	388,800	640,505	1.6474
20 Apr 2011 15:49:25	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	362,880	593,574	1.6357
20 Apr 2011 15:49:25	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	336,960	551,585	1.6369
20 Apr 2011 15:49:24	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	311,040	513,880	1.6521
20 Apr 2011 15:49:24	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	285,120	471,516	1.6537
12 Apr 2011 03:30:59	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	259,200	429,443	1.6568
11 Apr 2011 05:55:23	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	233,280	389,213	1.6684
10 Apr 2011 17:05:35	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	207,360	347,299	1.6749
10 Apr 2011 03:01:33	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	181,440	303,499	1.6727
09 Apr 2011 12:13:46	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	155,520	259,400	1.6680
08 Apr 2011 19:18:41	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	129,600	215,281	1.6611
07 Apr 2011 13:24:27	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	103,680	171,635	1.6554
04 Apr 2011 07:26:14	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	77,760	130,598	1.6795
03 Apr 2011 09:39:42	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	51,840	86,890	1.6761
02 Apr 2011 18:38:06	895482	12758764	hadcm3n_o42j_1900_40_007200606_2	25,920	42,861	1.6536