Task 19132617

Name	hadcm3n_xbjd_198012_480_010220674_1
Workunit	10220674
Created	4 Dec 2015, 17:10:47 UTC
Sent	4 Dec 2015, 18:11:53 UTC
Report deadline	15 Nov 2016, 23:31:53 UTC
Received	27 Dec 2015, 23:28:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1379272
Run time	4 days 8 hours 56 min 41 sec
CPU time	4 days 6 hours 28 min 51 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.85 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish 03:06:12 (2007): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:49:50 (28991): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:06:24 (9889): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:08:47 (10163): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:11:09 (10193): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:12:16 (10193): No heartbeat from core client for 30 sec - exiting 05:13:26 (10225): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:25:26 (10641): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:28:54 (10666): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish 21:30:02 (2038): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:44:41 (20615): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:52:58 (27844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:18:12 (28967): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:22:37 (4080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:26:30 (5260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:04 (6779): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:35:08 (7909): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:37:01 (8154): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish 02:23:44 (2036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:46:00 (11819): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:01:45 (18555): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:08:41 (22260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:19:42 (23797): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:18 (23797): No heartbeat from core client for 30 sec - exiting 03:20:49 (23797): No heartbeat from core client for 30 sec - exiting 03:28:07 (25714): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:21:21 (26509): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:03:30 (29594): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:04:05 (29594): No heartbeat from core client for 30 sec - exiting 05:04:43 (29594): No heartbeat from core client for 30 sec - exiting 05:04:44 (29594): No heartbeat from core client for 30 sec - exiting 05:04:45 (29594): No heartbeat from core client for 30 sec - exiting 05:04:46 (29594): No heartbeat from core client for 30 sec - exiting 05:04:47 (29594): No heartbeat from core client for 30 sec - exiting 05:07:13 (30529): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09ed3ea8 * ======= Backtrace: ========= /lib/libc.so.6(+0x6b0eb)[0xf74160eb] /lib/libc.so.6(+0x7401a)[0xf741f01a] /lib/libc.so.6(cfree+0x50)[0xf74226f0] /lib/libstdc++.so.6(_ZdlPv+0x1c)[0xf7653f2c] /lib/libstdc++.so.6(_ZdaPv+0x1c)[0xf7653fbc] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xf5)[0xf73c3545] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 fd:00 2363108 /var/lib/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 fd:00 2363108 /var/lib/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 09e79000-09ee4000 rw-p 00000000 00:00 0 [heap] f6e00000-f6e21000 rw-p 00000000 00:00 0 f6e21000-f6f00000 ---p 00000000 00:00 0 f6f2d000-f73a8000 rw-s 00000000 fd:00 2367886 /var/lib/boinc/slots/4/162730 f73a8000-f73ab000 rw-p 00000000 00:00 0 f73ab000-f7572000 r-xp 00000000 fd:00 792613 /usr/lib/libc-2.22.so f7572000-f7574000 r--p 001c7000 fd:00 792613 /usr/lib/libc-2.22.so f7574000-f7575000 rw-p 001c9000 fd:00 792613 /usr/lib/libc-2.22.so f7575000-f7578000 rw-p 00000000 00:00 0 f7578000-f7594000 r-xp 00000000 fd:00 793209 /usr/lib/libgcc_s-5.3.1-20151207.so.1 f7594000-f7595000 r--p 0001b000 fd:00 793209 /usr/lib/libgcc_s-5.3.1-20151207.so.1 f7595000-f7596000 rw-p 0001c000 fd:00 793209 /usr/lib/libgcc_s-5.3.1-20151207.so.1 f7596000-f7597000 rw-p 00000000 00:00 0 f7597000-f75e3000 r-xp 00000000 fd:00 792621 /usr/lib/libm-2.22.so f75e3000-f75e4000 r--p 0004c000 fd:00 792621 /usr/lib/libm-2.22.so f75e4000-f75e5000 rw-p 0004d000 fd:00 792621 /usr/lib/libm-2.22.so f75e5000-f7756000 r-xp 00000000 fd:00 793212 /usr/lib/libstdc++.so.6.0.21 f7756000-f775b000 r--p 00171000 fd:00 793212 /usr/lib/libstdc++.so.6.0.21 f775b000-f775c000 rw-p 00176000 fd:00 793212 /usr/lib/libstdc++.so.6.0.21 f775c000-f775f000 rw-p 00000000 00:00 0 f775f000-f7762000 r-xp 00000000 fd:00 792619 /usr/lib/libdl-2.22.so f7762000-f7763000 r--p 00002000 fd:00 792619 /usr/lib/libdl-2.22.so f7763000-f7764000 rw-p 00003000 fd:00 792619 /usr/lib/libdl-2.22.so f7764000-f777d000 r-xp 00000000 fd:00 792639 /usr/lib/libpthread-2.22.so f777d000-f777e000 r--p 00018000 fd:00 792639 /usr/lib/libpthread-2.22.so f777e000-f777f000 rw-p 00019000 fd:00 792639 /usr/lib/libpthread-2.22.so f777f000-f7781000 rw-p 00000000 00:00 0 f779e000-f77a0000 rw-p 00000000 00:00 0 f77a0000-f77a1000 ---p 00000000 00:00 0 f77a1000-f77a4000 rw-p 00000000 00:00 0 [stack:2386] f77a4000-f77a6000 rw-s 00000000 fd:00 2367615 /var/lib/boinc/slots/4/boinc_mmap_file f77a6000-f77a7000 rw-p 00000000 00:00 0 f77a7000-f77a9000 r--p 00000000 00:00 0 [vvar] f77a9000-f77aa000 r-xp 00000000 00:00 0 [vdso] f77aa000-f77cb000 r-xp 00000000 fd:00 792606 /usr/lib/ld-2.22.so f77cb000-f77cc000 r--p 00021000 fd:00 792606 /usr/lib/ld-2.22.so f77cc000-f77cd000 rw-p 00022000 fd:00 792606 /usr/lib/ld-2.22.so ffc11000-ffc80000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (18 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77a9b70] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf77a9ba0] /lib/libc.so.6(gsignal+0x46)[0xf73d8876] /lib/libc.so.6(abort+0x157)[0xf73da027] /lib/libc.so.6(__libc_fatal+0x0)[0xf74160f0] /lib/libc.so.6(+0x7401a)[0xf741f01a] /lib/libc.so.6(cfree+0x50)[0xf74226f0] /lib/libstdc++.so.6(_ZdlPv+0x1c)[0xf7653f2c] /lib/libstdc++.so.6(_ZdaPv+0x1c)[0xf7653fbc] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xf5)[0xf73c3545] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Dec 2015 23:33:11	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	518,400	368,928	0.7117
27 Dec 2015 18:27:37	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	492,480	350,956	0.7126
17 Dec 2015 07:24:20	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	466,560	332,915	0.7136
15 Dec 2015 03:25:16	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	440,640	314,616	0.7140
14 Dec 2015 22:29:35	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	414,720	296,525	0.7150
13 Dec 2015 17:48:57	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	388,800	278,458	0.7162
13 Dec 2015 10:06:46	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	362,880	260,493	0.7178
13 Dec 2015 04:33:21	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	336,960	242,517	0.7197
12 Dec 2015 23:04:25	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	311,040	224,426	0.7215
12 Dec 2015 18:07:41	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	285,120	206,512	0.7243
12 Dec 2015 13:00:26	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	259,200	188,566	0.7275
12 Dec 2015 07:57:27	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	233,280	170,608	0.7313
12 Dec 2015 01:55:42	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	207,360	152,719	0.7365
11 Dec 2015 21:09:38	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	181,440	134,799	0.7429
09 Dec 2015 05:43:37	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	155,520	116,862	0.7514
09 Dec 2015 01:39:53	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	129,600	98,989	0.7638
08 Dec 2015 19:33:15	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	103,680	80,947	0.7807
06 Dec 2015 06:32:33	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	77,760	63,020	0.8104
06 Dec 2015 01:06:51	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	51,840	44,251	0.8536
05 Dec 2015 20:05:08	1379272	19132617	hadcm3n_xbjd_198012_480_010220674_1	25,920	26,102	1.0070