Task 13351598

Name	hadcm3n_o7nv_1940_40_007444287_2
Workunit	7641790
Created	9 Sep 2011, 3:04:27 UTC
Sent	9 Sep 2011, 3:11:32 UTC
Report deadline	9 Dec 2011, 10:38:43 UTC
Received	26 Sep 2011, 5:28:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1317410
Run time	10 days 21 hours 7 min
CPU time	10 days 10 hours 36 min 35 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.54 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> 05:55:14 (13905): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:56:45 (26811): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:00:20 (26883): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:01:05 (26943): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:02:25 (26992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:03:11 (27045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:04:44 (27102): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:05:37 (27149): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:06:21 (27199): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:07:09 (27247): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:08:40 (27305): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:09:29 (27352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:10:58 (27410): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:11:46 (27459): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:13:16 (27518): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:14:16 (27565): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:15:00 (27614): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:17:08 (27681): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:38 (27743): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:19:25 (27791): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:20:56 (27848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:21:44 (27896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:23:14 (27956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:13 (28003): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:57 (28051): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:26:29 (28116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:28:36 (28182): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:29:23 (28230): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:30:16 (28277): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:31:46 (28337): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:33 (28384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:33:20 (28432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:34:56 (28490): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:35:39 (28537): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:36:35 (28585): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:37:23 (28633): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:38:10 (28682): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:39:03 (28729): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:39:54 (28777): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:41:37 (28836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:42:21 (28884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:43:10 (28932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:44:41 (28992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:45:26 (29039): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:46:12 (29086): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:47:57 (29146): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:48:45 (29196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:49:43 (29243): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:50:26 (29293): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:51:15 (29340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:52:12 (29387): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:53:42 (29446): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:54:30 (29495): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:56:02 (29557): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:56:47 (29605): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:57:36 (29651): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:59:06 (29709): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:59:54 (29757): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:01:23 (29817): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:02:10 (29864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:43 (29921): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:04:29 (29969): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:05:16 (30017): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:07:01 (30077): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:07:47 (30125): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:08:32 (30172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:10:12 (30231): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:10:56 (30279): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:11:50 (30328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:34 (30375): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:15:20 (30451): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:16:52 (30511): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:39 (30561): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:18:31 (30608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:20:02 (30666): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:20:48 (30714): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:21:35 (30763): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:24:27 (30840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:25:16 (30887): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:26:00 (30938): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:27:34 (30998): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:21 (31045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:29:18 (31092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:30:54 (31156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:39 (31206): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:32:26 (31254): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:33:17 (31301): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:34:48 (31359): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:35:34 (31433): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:14 (31506): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:38:01 (31556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:38:47 (31602): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:39:41 (31649): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:40:24 (31697): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:25:30 (6127): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:26:42 (6584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08f9bce8 *** ======= Backtrace: ========= /lib32/libc.so.6(+0x6b511)[0xf7549511] /lib32/libc.so.6(+0x6ce1b)[0xf754ae1b] /lib32/libc.so.6(cfree+0x6d)[0xf754df8d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7727841] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf772789d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf74f4e37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 08:05 794949 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:05 794949 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 08f46000-08fac000 rw-p 00000000 00:00 0 [heap] f6f00000-f6f21000 rw-p 00000000 00:00 0 f6f21000-f7000000 ---p 00000000 00:00 0 f7060000-f74db000 rw-s 00000000 08:05 803528 /var/lib/boinc-client/slots/9/138095 f74db000-f74de000 rw-p 00000000 00:00 0 f74de000-f7634000 r-xp 00000000 08:05 1700615 /lib32/libc-2.13.so f7634000-f7635000 ---p 00156000 08:05 1700615 /lib32/libc-2.13.so f7635000-f7637000 r--p 00156000 08:05 1700615 /lib32/libc-2.13.so f7637000-f7638000 rw-p 00158000 08:05 1700615 /lib32/libc-2.13.so f7638000-f763b000 rw-p 00000000 00:00 0 f763b000-f7655000 r-xp 00000000 08:05 277567 /usr/lib32/libgcc_s.so.1 f7655000-f7656000 r--p 00019000 08:05 277567 /usr/lib32/libgcc_s.so.1 f7656000-f7657000 rw-p 0001a000 08:05 277567 /usr/lib32/libgcc_s.so.1 f7657000-f767b000 r-xp 00000000 08:05 1700619 /lib32/libm-2.13.so f767b000-f767c000 r--p 00023000 08:05 1700619 /lib32/libm-2.13.so f767c000-f767d000 rw-p 00024000 08:05 1700619 /lib32/libm-2.13.so f767d000-f775c000 r-xp 00000000 08:05 277570 /usr/lib32/libstdc++.so.6.0.14 f775c000-f7760000 r--p 000de000 08:05 277570 /usr/lib32/libstdc++.so.6.0.14 f7760000-f7761000 rw-p 000e2000 08:05 277570 /usr/lib32/libstdc++.so.6.0.14 f7761000-f7768000 rw-p 00000000 00:00 0 f7768000-f776a000 r-xp 00000000 08:05 1700618 /lib32/libdl-2.13.so f776a000-f776b000 r--p 00001000 08:05 1700618 /lib32/libdl-2.13.so f776b000-f776c000 rw-p 00002000 08:05 1700618 /lib32/libdl-2.13.so f776c000-f7781000 r-xp 00000000 08:05 1700629 /lib32/libpthread-2.13.so f7781000-f7782000 r--p 00014000 08:05 1700629 /lib32/libpthread-2.13.so f7782000-f7783000 rw-p 00015000 08:05 1700629 /lib32/libpthread-2.13.so f7783000-f7786000 rw-p 00000000 00:00 0 f77a3000-f77a4000 rw-p 00000000 00:00 0 f77a4000-f77a5000 ---p 00000000 00:00 0 f77a5000-f77a8000 rw-p 00000000 00:00 0 f77a8000-f77aa000 rw-s 00000000 08:05 803076 /var/lib/boinc-client/slots/9/boinc_mmap_file f77aa000-f77ab000 rw-p 00000000 00:00 0 f77ab000-f77ac000 r-xp 00000000 00:00 0 [vdso] f77ac000-f77c8000 r-xp 00000000 08:05 1700612 /lib32/ld-2.13.so f77c8000-f77c9000 r--p 0001b000 08:05 1700612 /lib32/ld-2.13.so f77c9000-f77ca000 rw-p 0001c000 08:05 1700612 /lib32/ld-2.13.so ffa18000-ffa88000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf77ab400] [0xf77ab430] /lib32/libc.so.6(gsignal+0x51)[0xf7508ea1] /lib32/libc.so.6(abort+0x17e)[0xf750c2ce] /lib32/libc.so.6(+0x61277)[0xf753f277] /lib32/libc.so.6(+0x6b511)[0xf7549511] /lib32/libc.so.6(+0x6ce1b)[0xf754ae1b] /lib32/libc.so.6(cfree+0x6d)[0xf754df8d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7727841] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf772789d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf74f4e37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Sep 2011 04:27:07	1162363	13351598	hadcm3n_o7nv_1940_40_007444287_2	259,200	902,186	3.4807
25 Sep 2011 02:01:06	1162363	13351598	hadcm3n_o7nv_1940_40_007444287_2	233,280	811,838	3.4801
23 Sep 2011 23:55:38	1162363	13351598	hadcm3n_o7nv_1940_40_007444287_2	207,360	721,848	3.4811
22 Sep 2011 20:23:37	1162363	13351598	hadcm3n_o7nv_1940_40_007444287_2	181,440	632,336	3.4851
21 Sep 2011 18:36:54	1162363	13351598	hadcm3n_o7nv_1940_40_007444287_2	155,520	541,224	3.4801
20 Sep 2011 15:27:25	1162363	13351598	hadcm3n_o7nv_1940_40_007444287_2	129,600	451,045	3.4803
19 Sep 2011 12:15:31	1162363	13351598	hadcm3n_o7nv_1940_40_007444287_2	103,680	361,168	3.4835
18 Sep 2011 09:17:24	1162363	13351598	hadcm3n_o7nv_1940_40_007444287_2	77,760	270,629	3.4803
13 Sep 2011 07:01:18	1162363	13351598	hadcm3n_o7nv_1940_40_007444287_2	51,840	181,084	3.4931
11 Sep 2011 18:34:16	1162363	13351598	hadcm3n_o7nv_1940_40_007444287_2	25,920	91,190	3.5181