Task 15855932

Name	hadcm3n_o597_2060_40_008394021_0
Workunit	8544880
Created	22 Jun 2013, 1:15:51 UTC
Sent	22 Jun 2013, 15:40:07 UTC
Report deadline	21 Sep 2013, 23:07:18 UTC
Received	1 Jul 2013, 1:04:44 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1285400
Run time	5 days 13 hours 30 min 9 sec
CPU time	5 days 11 hours 54 min
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.45</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:09:24 (18572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:09:25 (18572): No heartbeat from core client for 30 sec - exiting 00:09:26 (18572): No heartbeat from core client for 30 sec - exiting 00:09:27 (18572): No heartbeat from core client for 30 sec - exiting 00:09:28 (18572): No heartbeat from core client for 30 sec - exiting 00:09:29 (18572): No heartbeat from core client for 30 sec - exiting 00:09:30 (18572): No heartbeat from core client for 30 sec - exiting 00:09:31 (18572): No heartbeat from core client for 30 sec - exiting 00:09:32 (18572): No heartbeat from core client for 30 sec - exiting 00:09:33 (18572): No heartbeat from core client for 30 sec - exiting 00:09:34 (18572): No heartbeat from core client for 30 sec - exiting 00:09:35 (18572): No heartbeat from core client for 30 sec - exiting 00:09:36 (18572): No heartbeat from core client for 30 sec - exiting 00:09:37 (18572): No heartbeat from core client for 30 sec - exiting 00:09:38 (18572): No heartbeat from core client for 30 sec - exiting 00:09:39 (18572): No heartbeat from core client for 30 sec - exiting 00:09:40 (18572): No heartbeat from core client for 30 sec - exiting 00:09:41 (18572): No heartbeat from core client for 30 sec - exiting 00:09:42 (18572): No heartbeat from core client for 30 sec - exiting 00:09:43 (18572): No heartbeat from core client for 30 sec - exiting 00:09:44 (18572): No heartbeat from core client for 30 sec - exiting 00:09:45 (18572): No heartbeat from core client for 30 sec - exiting 00:09:46 (18572): No heartbeat from core client for 30 sec - exiting 00:09:47 (18572): No heartbeat from core client for 30 sec - exiting 00:09:48 (18572): No heartbeat from core client for 30 sec - exiting 00:09:49 (18572): No heartbeat from core client for 30 sec - exiting 00:09:50 (18572): No heartbeat from core client for 30 sec - exiting 00:09:51 (18572): No heartbeat from core client for 30 sec - exiting Signal 1 received, exiting... Called boinc_finish 00:45:56 (1455): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:52:05 (21431): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:06:54 (7147): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:30:59 (32092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0949d6f0 *** ======= Backtrace: ========= /lib/libc.so.6(+0x70e31)[0x216e31] /lib/libc.so.6(+0x73571)[0x219571] /usr/lib/libstdc++.so.6(_ZdlPv+0x22)[0xb3bdf2] /usr/lib/libstdc++.so.6(_ZdaPv+0x1e)[0xb3be4e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xe6)[0x1bcce6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 001a6000-00336000 r-xp 00000000 fd:00 138123 /lib/libc-2.12.so 00336000-00337000 ---p 00190000 fd:00 138123 /lib/libc-2.12.so 00337000-00339000 r--p 00190000 fd:00 138123 /lib/libc-2.12.so 00339000-0033a000 rw-p 00192000 fd:00 138123 /lib/libc-2.12.so 0033a000-0033d000 rw-p 00000000 00:00 0 005e7000-0060f000 r-xp 00000000 fd:00 138138 /lib/libm-2.12.so 0060f000-00610000 r--p 00027000 fd:00 138138 /lib/libm-2.12.so 00610000-00611000 rw-p 00028000 fd:00 138138 /lib/libm-2.12.so 00619000-00630000 r-xp 00000000 fd:00 139787 /lib/libpthread-2.12.so 00630000-00631000 r--p 00016000 fd:00 139787 /lib/libpthread-2.12.so 00631000-00632000 rw-p 00017000 fd:00 139787 /lib/libpthread-2.12.so 00632000-00634000 rw-p 00000000 00:00 0 00675000-00692000 r-xp 00000000 fd:00 136304 /lib/libgcc_s-4.4.7-20120601.so.1 00692000-00693000 rw-p 0001d000 fd:00 136304 /lib/libgcc_s-4.4.7-20120601.so.1 009d1000-009d2000 r-xp 00000000 00:00 0 [vdso] 00a8f000-00b6e000 r-xp 00000000 fd:00 37676 /usr/lib/libstdc++.so.6.0.13 00b6e000-00b72000 r--p 000de000 fd:00 37676 /usr/lib/libstdc++.so.6.0.13 00b72000-00b73000 rw-p 000e2000 fd:00 37676 /usr/lib/libstdc++.so.6.0.13 00b73000-00b7a000 rw-p 00000000 00:00 0 00d1f000-00d3d000 r-xp 00000000 fd:00 137181 /lib/ld-2.12.so 00d3d000-00d3e000 r--p 0001d000 fd:00 137181 /lib/ld-2.12.so 00d3e000-00d3f000 rw-p 0001e000 fd:00 137181 /lib/ld-2.12.so 00fc8000-00fcb000 r-xp 00000000 fd:00 138136 /lib/libdl-2.12.so 00fcb000-00fcc000 r--p 00002000 fd:00 138136 /lib/libdl-2.12.so 00fcc000-00fcd000 rw-p 00003000 fd:00 138136 /lib/libdl-2.12.so 08048000-080e3000 r-xp 00000000 fd:00 146006 /var/lib/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 fd:00 146006 /var/lib/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 09448000-094ce000 rw-p 00000000 00:00 0 [heap] f7100000-f7121000 rw-p 00000000 00:00 0 f7121000-f7200000 ---p 00000000 00:00 0 f727b000-f76f6000 rw-s 00000000 fd:00 148619 /var/lib/boinc/slots/10/130115 f76f6000-f76f9000 rw-p 00000000 00:00 0 f76ff000-f7700000 rw-p 00000000 00:00 0 f7700000-f7701000 ---p 00000000 00:00 0 f7701000-f7704000 rw-p 00000000 00:00 0 f7704000-f7706000 rw-s 00000000 fd:00 148465 /var/lib/boinc/slots/10/boinc_mmap_file f7706000-f7708000 rw-p 00000000 00:00 0 fff15000-fff84000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (18 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x9d1400] [0x9d1430] /lib/libc.so.6(gsignal+0x51)[0x1d0b11] /lib/libc.so.6(abort+0x17a)[0x1d23ea] /lib/libc.so.6(+0x6a9d5)[0x2109d5] /lib/libc.so.6(+0x70e31)[0x216e31] /lib/libc.so.6(+0x73571)[0x219571] /usr/lib/libstdc++.so.6(_ZdlPv+0x22)[0xb3bdf2] /usr/lib/libstdc++.so.6(_ZdaPv+0x1e)[0xb3be4e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xe6)[0x1bcce6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jul 2013 11:06:49	1285400	15855932	hadcm3n_o597_2060_40_008394021_0	259,200	474,833	1.8319
02 Jul 2013 10:28:38	1285400	15855932	hadcm3n_o597_2060_40_008394021_0	233,280	426,081	1.8265
02 Jul 2013 09:59:46	1285400	15855932	hadcm3n_o597_2060_40_008394021_0	207,360	378,292	1.8243
28 Jun 2013 12:04:59	1285400	15855932	hadcm3n_o597_2060_40_008394021_0	181,440	331,080	1.8247
27 Jun 2013 15:41:57	1285400	15855932	hadcm3n_o597_2060_40_008394021_0	155,520	283,570	1.8234
26 Jun 2013 19:46:00	1285400	15855932	hadcm3n_o597_2060_40_008394021_0	129,600	235,412	1.8165
26 Jun 2013 01:45:58	1285400	15855932	hadcm3n_o597_2060_40_008394021_0	103,680	188,311	1.8163
25 Jun 2013 08:15:57	1285400	15855932	hadcm3n_o597_2060_40_008394021_0	77,760	141,694	1.8222
24 Jun 2013 13:37:06	1285400	15855932	hadcm3n_o597_2060_40_008394021_0	51,840	96,076	1.8533
23 Jun 2013 14:39:17	1285400	15855932	hadcm3n_o597_2060_40_008394021_0	25,920	48,502	1.8712