Task 15934905

Name	hadcm3n_n3bt_1920_40_008410479_0
Workunit	8561335
Created	22 Aug 2013, 5:39:13 UTC
Sent	22 Aug 2013, 6:06:44 UTC
Report deadline	21 Nov 2013, 13:33:55 UTC
Received	5 Oct 2013, 13:30:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1110299
Run time	28 days 20 hours 54 min 11 sec
CPU time	23 days 22 hours 23 min 29 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.92 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/n3btko.pjc4c10 is not a valid UM file. Error converting file to netcdf: dataout/n3btko.pjc4c10 Error: Input file: dataout/n3btko.pic4c10 is not a valid UM file. Error converting file to netcdf: dataout/n3btko.pic4c10 Error: Input file: dataout/n3btko.pfc4c10 is not a valid UM file. Error converting file to netcdf: dataout/n3btko.pfc4c10 Error: Input file: dataout/n3btka.phc4c10 is not a valid UM file. Error converting file to netcdf: dataout/n3btka.phc4c10 Error: Input file: dataout/n3btka.pgc4c10 is not a valid UM file. Error converting file to netcdf: dataout/n3btka.pgc4c10 Error: Input file: dataout/n3btka.pec4c10 is not a valid UM file. Error converting file to netcdf: dataout/n3btka.pec4c10 Error: Input file: dataout/n3btka.pdc4c10 is not a valid UM file. Error converting file to netcdf: dataout/n3btka.pdc4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 01:12:51 (91079): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:38:38 (1713): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:26:10 (404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:15:54 (781): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:15:55 (781): No heartbeat from core client for 30 sec - exiting 15:15:56 (781): No heartbeat from core client for 30 sec - exiting 15:15:57 (781): No heartbeat from core client for 30 sec - exiting 15:15:58 (781): No heartbeat from core client for 30 sec - exiting 15:15:59 (781): No heartbeat from core client for 30 sec - exiting 15:16:00 (781): No heartbeat from core client for 30 sec - exiting 15:16:01 (781): No heartbeat from core client for 30 sec - exiting 15:16:02 (781): No heartbeat from core client for 30 sec - exiting 17:05:21 (7650): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:05:23 (7650): No heartbeat from core client for 30 sec - exiting 17:05:24 (7650): No heartbeat from core client for 30 sec - exiting 20:33:48 (9540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:00:05 (13698): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:07:25 (30323): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:08:49 (33430): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:31:48 (34563): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:03:10 (42793): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:13:20 (61602): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:35:41 (75292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:35:42 (75292): No heartbeat from core client for 30 sec - exiting 21:35:43 (75292): No heartbeat from core client for 30 sec - exiting 21:35:44 (75292): No heartbeat from core client for 30 sec - exiting 21:35:45 (75292): No heartbeat from core client for 30 sec - exiting 21:35:46 (75292): No heartbeat from core client for 30 sec - exiting 21:35:47 (75292): No heartbeat from core client for 30 sec - exiting 21:35:48 (75292): No heartbeat from core client for 30 sec - exiting 21:35:49 (75292): No heartbeat from core client for 30 sec - exiting 21:35:50 (75292): No heartbeat from core client for 30 sec - exiting 06:28:44 (99189): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:22:58 (8164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:14:33 (12064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:14:34 (12064): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 10:42:39 (20497): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:42:40 (20497): No heartbeat from core client for 30 sec - exiting 10:42:41 (20497): No heartbeat from core client for 30 sec - exiting 10:42:42 (20497): No heartbeat from core client for 30 sec - exiting 10:42:43 (20497): No heartbeat from core client for 30 sec - exiting 10:42:44 (20497): No heartbeat from core client for 30 sec - exiting 10:42:45 (20497): No heartbeat from core client for 30 sec - exiting 10:42:46 (20497): No heartbeat from core client for 30 sec - exiting 10:42:47 (20497): No heartbeat from core client for 30 sec - exiting 10:42:48 (20497): No heartbeat from core client for 30 sec - exiting 10:42:49 (20497): No heartbeat from core client for 30 sec - exiting 10:42:50 (20497): No heartbeat from core client for 30 sec - exiting 10:42:51 (20497): No heartbeat from core client for 30 sec - exiting 10:42:52 (20497): No heartbeat from core client for 30 sec - exiting 10:42:53 (20497): No heartbeat from core client for 30 sec - exiting 10:42:54 (20497): No heartbeat from core client for 30 sec - exiting 10:42:55 (20497): No heartbeat from core client for 30 sec - exiting 10:42:56 (20497): No heartbeat from core client for 30 sec - exiting 10:42:57 (20497): No heartbeat from core client for 30 sec - exiting 10:42:58 (20497): No heartbeat from core client for 30 sec - exiting 12:22:38 (40077): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:22:40 (40077): No heartbeat from core client for 30 sec - exiting 14:08:13 (41727): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:38:25 (43505): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:39:47 (45979): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:55 (47048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:56 (47048): No heartbeat from core client for 30 sec - exiting 16:22:57 (47048): No heartbeat from core client for 30 sec - exiting 16:22:58 (47048): No heartbeat from core client for 30 sec - exiting 16:22:59 (47048): No heartbeat from core client for 30 sec - exiting 16:23:00 (47048): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 22:21:43 (72712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:14:43 (78704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:11:12 (86995): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:13:23 (99241): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:08:44 (1608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:08:45 (1608): No heartbeat from core client for 30 sec - exiting 07:08:46 (1608): No heartbeat from core client for 30 sec - exiting 07:08:47 (1608): No heartbeat from core client for 30 sec - exiting 07:08:48 (1608): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:58:11 (8051): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(26424,0xacb492c0) malloc: * error for object 0x60b4e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26424,0xacb492c0) malloc: * error for object 0x60b4e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26424,0xacb492c0) malloc: * error for object 0x60b4e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26424,0xacb492c0) malloc: * error for object 0x60b4e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26424,0xacb492c0) malloc: * error for object 0x60b4e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26424,0xacb492c0) malloc: * error for object 0x60b4e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26424,0xacb492c0) malloc: * error for object 0x60b4e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(41273,0xacb492c0) malloc: * error for object 0x688ea04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(41273,0xacb492c0) malloc: * error for object 0x688ea00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(41273,0xacb492c0) malloc: * error for object 0x3086c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.5 build 11G63 Sat Oct 5 12:20:45 2013 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fd88 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/44/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x908a6000 - 0x908aefff /usr/lib/system/libcopyfile.dylib 0x91cbf000 - 0x91cc7fff /usr/lib/system/liblaunch.dylib 0x91f2f000 - 0x91f30fff /usr/lib/system/libremovefile.dylib 0x927d1000 - 0x927d4fff /usr/lib/system/libmathCommon.A.dylib 0x9317d000 - 0x9317efff /usr/lib/system/libunc.dylib 0x9371f000 - 0x9374efff /usr/lib/system/libsystem_info.dylib 0x94b8f000 - 0x94b92fff /usr/lib/system/libcompiler_rt.dylib 0x94cd8000 - 0x94ceefff /usr/lib/system/libxpc.dylib 0x9500b000 - 0x9500dfff /usr/lib/system/libdyld.dylib 0x9501a000 - 0x95022fff /usr/lib/system/libunwind.dylib 0x95b49000 - 0x95b49fff /usr/lib/system/libkeymgr.dylib 0x95b4a000 - 0x95b51fff /usr/lib/system/libsystem_notify.dylib 0x95bd4000 - 0x95bf2fff /usr/lib/system/libsystem_kernel.dylib 0x973d0000 - 0x973d1fff /usr/lib/system/libquarantine.dylib 0x97487000 - 0x9748cfff /usr/lib/system/libmacho.dylib 0x9748d000 - 0x9748dfff /usr/lib/system/libdnsinfo.dylib 0x9905a000 - 0x99088fff /usr/lib/libSystem.B.dylib 0x99089000 - 0x99097fff /usr/lib/system/libdispatch.dylib 0x9944e000 - 0x99519fff /usr/lib/system/libsystem_c.dylib 0x99c9f000 - 0x99ce2fff /usr/lib/system/libcommonCrypto.dylib 0x99dc0000 - 0x99dc9fff /usr/lib/libc++abi.dylib 0x99dca000 - 0x99dcbfff /usr/lib/system/libsystem_sandbox.dylib 0x99dcc000 - 0x99ddafff /usr/lib/libz.1.dylib 0x9a2c7000 - 0x9a2cefff /usr/lib/system/libsystem_dnssd.dylib 0x9a5d4000 - 0x9a5d5fff /usr/lib/system/libsystem_blocks.dylib 0x9a692000 - 0x9a696fff /usr/lib/system/libcache.dylib 0x9ab21000 - 0x9ab25fff /usr/lib/system/libsystem_network.dylib 0x9cae1000 - 0x9cb43fff /usr/lib/libstdc++.6.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Oct 2013 08:58:57	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	1,036,800	2,067,060	1.9937
04 Oct 2013 17:05:10	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	1,010,880	2,019,646	1.9979
03 Oct 2013 19:39:30	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	984,960	1,977,027	2.0072
03 Oct 2013 05:19:43	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	959,040	1,931,825	2.0143
02 Oct 2013 01:02:54	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	933,120	1,881,115	2.0159
01 Oct 2013 04:08:42	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	907,200	1,827,963	2.0150
30 Sep 2013 05:23:35	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	881,280	1,776,605	2.0159
28 Sep 2013 23:25:50	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	855,360	1,727,968	2.0202
26 Sep 2013 12:23:49	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	829,440	1,677,897	2.0229
25 Sep 2013 09:28:56	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	803,520	1,625,750	2.0233
24 Sep 2013 04:32:08	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	777,600	1,571,655	2.0212
23 Sep 2013 09:53:19	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	751,680	1,519,457	2.0214
23 Sep 2013 09:53:19	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	725,760	1,469,421	2.0247
21 Sep 2013 08:25:28	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	699,840	1,419,041	2.0277
20 Sep 2013 13:36:24	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	673,920	1,368,285	2.0303
19 Sep 2013 12:23:14	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	648,000	1,319,500	2.0363
18 Sep 2013 14:42:33	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	622,080	1,269,329	2.0405
17 Sep 2013 17:12:52	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	596,160	1,218,867	2.0445
16 Sep 2013 20:40:01	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	570,240	1,167,050	2.0466
16 Sep 2013 01:13:55	1110299	15934905	hadcm3n_n3bt_1920_40_008410479_0	544,320	1,116,960	2.0520