Task 13336863

Name	hadcm3n_o49a_1900_40_007440012_0
Workunit	7637515
Created	5 Sep 2011, 18:18:36 UTC
Sent	5 Sep 2011, 23:52:38 UTC
Report deadline	6 Dec 2011, 7:19:49 UTC
Received	3 Nov 2011, 22:38:06 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1178154
Run time	20 days 23 hours 59 min
CPU time	18 days 16 hours 29 min 17 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.21 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 3 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1979, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /data/home/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb78c6400] /data/home/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x82e12bf] /data/home/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838e68c] /data/home/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x82dc8ea] /data/home/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x829a4a7] /data/home/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821c545] /data/home/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8219da1] /data/home/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /data/home/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /data/home/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /data/home/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb7705113] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1233, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (!prev): 0x08ecd0e8 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x6ebc2)[0xb7494bc2] /lib/i386-linux-gnu/libc.so.6(+0x6f862)[0xb7495862] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0xb749894d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0xb769780f] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1b)[0xb769786b] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb743f113] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 09:00 7471644 /data/home/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 09:00 7471644 /data/home/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 08e09000-08f09000 rw-p 00000000 00:00 0 [heap] b6e00000-b6e21000 rw-p 00000000 00:00 0 b6e21000-b6f00000 ---p 00000000 00:00 0 b6fa8000-b7423000 rw-s 00000000 09:00 11010580 /data/home/boinc/slots/0/131360 b7423000-b7426000 rw-p 00000000 00:00 0 b7426000-b759c000 r-xp 00000000 08:04 3802021 /lib/i386-linux-gnu/libc-2.13.so b759c000-b759e000 r--p 00176000 08:04 3802021 /lib/i386-linux-gnu/libc-2.13.so b759e000-b759f000 rw-p 00178000 08:04 3802021 /lib/i386-linux-gnu/libc-2.13.so b759f000-b75a2000 rw-p 00000000 00:00 0 b75a2000-b75be000 r-xp 00000000 08:04 3802042 /lib/i386-linux-gnu/libgcc_s.so.1 b75be000-b75bf000 r--p 0001b000 08:04 3802042 /lib/i386-linux-gnu/libgcc_s.so.1 b75bf000-b75c0000 rw-p 0001c000 08:04 3802042 /lib/i386-linux-gnu/libgcc_s.so.1 b75c0000-b75e8000 r-xp 00000000 08:04 3802051 /lib/i386-linux-gnu/libm-2.13.so b75e8000-b75e9000 r--p 00028000 08:04 3802051 /lib/i386-linux-gnu/libm-2.13.so b75e9000-b75ea000 rw-p 00029000 08:04 3802051 /lib/i386-linux-gnu/libm-2.13.so b75ea000-b76c8000 r-xp 00000000 08:04 4199834 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 b76c8000-b76c9000 ---p 000de000 08:04 4199834 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 b76c9000-b76cd000 r--p 000de000 08:04 4199834 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 b76cd000-b76ce000 rw-p 000e2000 08:04 4199834 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 b76ce000-b76d6000 rw-p 00000000 00:00 0 b76d6000-b76d9000 r-xp 00000000 08:04 3802032 /lib/i386-linux-gnu/libdl-2.13.so b76d9000-b76da000 r--p 00002000 08:04 3802032 /lib/i386-linux-gnu/libdl-2.13.so b76da000-b76db000 rw-p 00003000 08:04 3802032 /lib/i386-linux-gnu/libdl-2.13.so b76db000-b76f2000 r-xp 00000000 08:04 3802081 /lib/i386-linux-gnu/libpthread-2.13.so b76f2000-b76f3000 r--p 00016000 08:04 3802081 /lib/i386-linux-gnu/libpthread-2.13.so b76f3000-b76f4000 rw-p 00017000 08:04 3802081 /lib/i386-linux-gnu/libpthread-2.13.so b76f4000-b76f6000 rw-p 00000000 00:00 0 b7703000-b7704000 rw-p 00000000 00:00 0 b7704000-b7705000 ---p 00000000 00:00 0 b7705000-b7708000 rw-p 00000000 00:00 0 b7708000-b770a000 rw-s 00000000 09:00 11010577 /data/home/boinc/slots/0/boinc_mmap_file b770a000-b770c000 rw-p 00000000 00:00 0 b770c000-b770d000 r-xp 00000000 00:00 0 [vdso] b770d000-b772b000 r-xp 00000000 08:04 3802008 /lib/i386-linux-gnu/ld-2.13.so b772b000-b772c000 r--p 0001d000 08:04 3802008 /lib/i386-linux-gnu/ld-2.13.so b772c000-b772d000 rw-p 0001e000 08:04 3802008 /lib/i386-linux-gnu/ld-2.13.so bfe24000-bfe95000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb770c400] [0xb770c424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb7453c8f] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb74572b5] /lib/i386-linux-gnu/libc.so.6(+0x63dfc)[0xb7489dfc] /lib/i386-linux-gnu/libc.so.6(+0x6ebc2)[0xb7494bc2] /lib/i386-linux-gnu/libc.so.6(+0x6f862)[0xb7495862] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0xb749894d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0xb769780f] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1b)[0xb769786b] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb743f113] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Nov 2011 22:43:13	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	1,036,800	1,614,554	1.5572
03 Nov 2011 04:17:12	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	1,010,880	1,569,396	1.5525
02 Nov 2011 15:14:52	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	984,960	1,524,070	1.5473
02 Nov 2011 01:50:04	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	959,040	1,477,895	1.5410
01 Nov 2011 12:52:21	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	933,120	1,432,659	1.5353
31 Oct 2011 23:31:29	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	907,200	1,389,097	1.5312
31 Oct 2011 19:44:21	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	881,280	1,347,578	1.5291
31 Oct 2011 19:27:57	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	855,360	1,306,062	1.5269
31 Oct 2011 19:08:27	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	829,440	1,304,191	1.5724
31 Oct 2011 18:40:53	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	803,520	1,261,699	1.5702
31 Oct 2011 18:23:14	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	777,600	1,219,928	1.5688
31 Oct 2011 17:39:42	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	751,680	1,181,030	1.5712
31 Oct 2011 17:39:42	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	725,760	1,140,506	1.5715
18 Oct 2011 22:58:52	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	699,840	1,104,749	1.5786
18 Oct 2011 05:20:48	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	673,920	1,074,118	1.5938
17 Oct 2011 12:19:55	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	648,000	1,043,423	1.6102
16 Oct 2011 19:27:59	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	622,080	1,012,730	1.6280
16 Oct 2011 10:48:33	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	596,160	982,739	1.6484
16 Oct 2011 02:01:56	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	570,240	952,296	1.6700
14 Oct 2011 19:33:32	1160803	13336863	hadcm3n_o49a_1900_40_007440012_0	544,320	910,021	1.6718