Task 16079592

Name	hadcm3n_859u_1980_40_008464790_4
Workunit	8615629
Created	12 Nov 2013, 0:07:50 UTC
Sent	12 Nov 2013, 0:07:50 UTC
Report deadline	11 Feb 2014, 7:35:01 UTC
Received	27 Nov 2013, 15:12:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1171203
Run time	6 days 22 hours 8 min 45 sec
CPU time	5 days 23 hours 5 min 4 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.08 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:00:33 (13421): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:01 (13485): No heartbeat from core client for 30 sec - exiting 18:02:05 (13485): No heartbeat from core client for 30 sec - exiting 18:02:06 (13485): No heartbeat from core client for 30 sec - exiting 18:02:07 (13485): No heartbeat from core client for 30 sec - exiting 18:02:08 (13485): No heartbeat from core client for 30 sec - exiting 18:02:09 (13485): No heartbeat from core client for 30 sec - exiting 18:02:10 (13485): No heartbeat from core client for 30 sec - exiting 18:02:11 (13485): No heartbeat from core client for 30 sec - exiting 18:02:12 (13485): No heartbeat from core client for 30 sec - exiting 18:02:13 (13485): No heartbeat from core client for 30 sec - exiting 18:02:14 (13485): No heartbeat from core client for 30 sec - exiting 18:02:15 (13485): No heartbeat from core client for 30 sec - exiting 18:02:16 (13485): No heartbeat from core client for 30 sec - exiting 18:02:17 (13485): No heartbeat from core client for 30 sec - exiting 18:02:18 (13485): No heartbeat from core client for 30 sec - exiting 18:02:19 (13485): No heartbeat from core client for 30 sec - exiting 18:02:20 (13485): No heartbeat from core client for 30 sec - exiting 18:02:21 (13485): No heartbeat from core client for 30 sec - exiting 18:02:22 (13485): No heartbeat from core client for 30 sec - exiting 18:02:23 (13485): No heartbeat from core client for 30 sec - exiting 18:02:24 (13485): No heartbeat from core client for 30 sec - exiting 18:02:26 (13485): No heartbeat from core client for 30 sec - exiting 18:02:27 (13485): No heartbeat from core client for 30 sec - exiting 18:02:28 (13485): No heartbeat from core client for 30 sec - exiting 18:02:29 (13485): No heartbeat from core client for 30 sec - exiting 18:02:30 (13485): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:04:08 (13571): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:04:10 (13571): No heartbeat from core client for 30 sec - exiting 18:05:35 (13617): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:37 (13682): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:00:32 (16990): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:01:34 (17041): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:04:28 (17099): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:20 (17149): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:29 (17197): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:15:16 (17254): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:16:05 (17254): No heartbeat from core client for 30 sec - exiting 18:19:36 (17451): No heartbeat from core client for 30 sec - exiting 18:19:39 (17451): No heartbeat from core client for 30 sec - exiting 18:19:40 (17451): No heartbeat from core client for 30 sec - exiting 18:19:41 (17451): No heartbeat from core client for 30 sec - exiting 18:19:42 (17451): No heartbeat from core client for 30 sec - exiting 18:19:43 (17451): No heartbeat from core client for 30 sec - exiting 18:19:45 (17451): No heartbeat from core client for 30 sec - exiting 18:19:46 (17451): No heartbeat from core client for 30 sec - exiting 18:19:47 (17451): No heartbeat from core client for 30 sec - exiting 18:19:48 (17451): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:23:31 (17490): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:34 (17521): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:16 (17565): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:39 (17665): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:42 (17717): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:42:20 (17791): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:48:40 (17906): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:03:22 (18028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:05:05 (18228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:56 (18277): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:57 (18277): No heartbeat from core client for 30 sec - exiting 19:12:30 (18330): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:25:28 (18412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:26:25 (18551): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:38 (18635): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:35:52 (18719): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:35:54 (18719): No heartbeat from core client for 30 sec - exiting 19:35:55 (18719): No heartbeat from core client for 30 sec - exiting 19:35:56 (18719): No heartbeat from core client for 30 sec - exiting 19:35:57 (18719): No heartbeat from core client for 30 sec - exiting 19:35:58 (18719): No heartbeat from core client for 30 sec - exiting 19:37:12 (18765): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:45:12 (18829): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:45:15 (18829): No heartbeat from core client for 30 sec - exiting 19:53:15 (19044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:09:05 (19167): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:33 (19432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:13:09 (19541): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:38 (19603): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:35:04 (19685): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:38:01 (19970): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:38:06 (19970): No heartbeat from core client for 30 sec - exiting 20:38:07 (19970): No heartbeat from core client for 30 sec - exiting 20:38:08 (19970): No heartbeat from core client for 30 sec - exiting 20:42:38 (20008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:45:12 (20088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:51:55 (20253): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:55:39 (20349): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:59:10 (20411): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:02:32 (20469): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:13:28 (20547): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:17:43 (20722): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:20:13 (20795): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:24:19 (20875): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:28:26 (20956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:20 (21028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:38:13 (21343): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:36:30 (16195): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:33 (16195): No heartbeat from core client for 30 sec - exiting 11:36:34 (16195): No heartbeat from core client for 30 sec - exiting 11:36:35 (16195): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:50:29 (29860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:50:30 (29860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 18:00:31 (12023): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:00:33 (12023): No heartbeat from core client for 30 sec - exiting 18:02:03 (12072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:04:27 (12121): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:42 (12166): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:17 (12204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:22:32 (12265): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (!prev): 0x098cb948 *** ======= Backtrace: ========= /lib32/libc.so.6(+0x6efe1)[0xf7488fe1] /lib32/libc.so.6(+0x70868)[0xf748a868] /lib32/libc.so.6(cfree+0x6d)[0xf748d9ad] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf767ba91] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf767baed] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7430bf6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 09:01 262689 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 09:01 262689 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 0987c000-098e1000 rw-p 00000000 00:00 0 [heap] f6e00000-f6e21000 rw-p 00000000 00:00 0 f6e21000-f6f00000 ---p 00000000 00:00 0 f6f9d000-f7418000 rw-s 00000000 09:01 131629 /var/lib/boinc-client/slots/3/133565 f7418000-f741a000 rw-p 00000000 00:00 0 f741a000-f7574000 r-xp 00000000 09:01 262916 /lib32/libc-2.11.1.so f7574000-f7575000 ---p 0015a000 09:01 262916 /lib32/libc-2.11.1.so f7575000-f7577000 r--p 0015a000 09:01 262916 /lib32/libc-2.11.1.so f7577000-f7578000 rw-p 0015c000 09:01 262916 /lib32/libc-2.11.1.so f7578000-f757b000 rw-p 00000000 00:00 0 f757b000-f7598000 r-xp 00000000 09:01 821291 /usr/lib32/libgcc_s.so.1 f7598000-f7599000 r--p 0001c000 09:01 821291 /usr/lib32/libgcc_s.so.1 f7599000-f759a000 rw-p 0001d000 09:01 821291 /usr/lib32/libgcc_s.so.1 f759a000-f75be000 r-xp 00000000 09:01 262921 /lib32/libm-2.11.1.so f75be000-f75bf000 r--p 00023000 09:01 262921 /lib32/libm-2.11.1.so f75bf000-f75c0000 rw-p 00024000 09:01 262921 /lib32/libm-2.11.1.so f75c0000-f76a9000 r-xp 00000000 09:01 821289 /usr/lib32/libstdc++.so.6.0.13 f76a9000-f76ad000 r--p 000e9000 09:01 821289 /usr/lib32/libstdc++.so.6.0.13 f76ad000-f76ae000 rw-p 000ed000 09:01 821289 /usr/lib32/libstdc++.so.6.0.13 f76ae000-f76b5000 rw-p 00000000 00:00 0 f76b5000-f76b7000 r-xp 00000000 09:01 262911 /lib32/libdl-2.11.1.so f76b7000-f76b8000 r--p 00001000 09:01 262911 /lib32/libdl-2.11.1.so f76b8000-f76b9000 rw-p 00002000 09:01 262911 /lib32/libdl-2.11.1.so f76b9000-f76ba000 rw-p 00000000 00:00 0 f76ba000-f76cf000 r-xp 00000000 09:01 262907 /lib32/libpthread-2.11.1.so f76cf000-f76d0000 r--p 00014000 09:01 262907 /lib32/libpthread-2.11.1.so f76d0000-f76d1000 rw-p 00015000 09:01 262907 /lib32/libpthread-2.11.1.so f76d1000-f76d3000 rw-p 00000000 00:00 0 f76e7000-f76e8000 rw-p 00000000 00:00 0 f76e8000-f76e9000 ---p 00000000 00:00 0 f76e9000-f76ec000 rw-p 00000000 00:00 0 f76ec000-f76ee000 rw-s 00000000 09:01 148554 /var/lib/boinc-client/slots/3/boinc_mmap_file f76ee000-f76f0000 rw-p 00000000 00:00 0 f76f0000-f76f1000 r-xp 00000000 00:00 0 [vdso] f76f1000-f770d000 r-xp 00000000 09:01 262912 /lib32/ld-2.11.1.so f770d000-f770e000 r--p 0001b000 09:01 262912 /lib32/ld-2.11.1.so f770e000-f770f000 rw-p 0001c000 09:01 262912 /lib32/ld-2.11.1.so ffd24000-ffd94000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf76f0400] [0xf76f0430] /lib32/libc.so.6(gsignal+0x51)[0xf7444971] /lib32/libc.so.6(abort+0x182)[0xf7447da2] /lib32/libc.so.6(+0x64eed)[0xf747eeed] /lib32/libc.so.6(+0x6efe1)[0xf7488fe1] /lib32/libc.so.6(+0x70868)[0xf748a868] /lib32/libc.so.6(cfree+0x6d)[0xf748d9ad] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf767ba91] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf767baed] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7430bf6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Nov 2013 15:18:39	1171203	16079592	hadcm3n_859u_1980_40_008464790_4	259,200	515,100	1.9873
26 Nov 2013 13:55:23	1171203	16079592	hadcm3n_859u_1980_40_008464790_4	233,280	463,950	1.9888
25 Nov 2013 14:40:41	1171203	16079592	hadcm3n_859u_1980_40_008464790_4	207,360	414,631	1.9996
24 Nov 2013 14:53:04	1171203	16079592	hadcm3n_859u_1980_40_008464790_4	181,440	359,129	1.9793
23 Nov 2013 20:08:44	1171203	16079592	hadcm3n_859u_1980_40_008464790_4	155,520	313,390	2.0151
23 Nov 2013 02:50:53	1171203	16079592	hadcm3n_859u_1980_40_008464790_4	129,600	261,678	2.0191
22 Nov 2013 09:11:20	1171203	16079592	hadcm3n_859u_1980_40_008464790_4	103,680	205,848	1.9854
17 Nov 2013 16:59:31	1171203	16079592	hadcm3n_859u_1980_40_008464790_4	77,760	166,556	2.1419
14 Nov 2013 06:57:53	1171203	16079592	hadcm3n_859u_1980_40_008464790_4	51,840	111,909	2.1587
12 Nov 2013 15:53:09	1171203	16079592	hadcm3n_859u_1980_40_008464790_4	25,920	52,040	2.0077