Task 13408518

Name	hadcm3n_u1n2_1980_40_007459056_0
Workunit	7656559
Created	22 Sep 2011, 17:35:35 UTC
Sent	22 Sep 2011, 17:45:50 UTC
Report deadline	23 Dec 2011, 1:13:01 UTC
Received	14 Oct 2011, 7:57:26 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	987797
Run time	7 days 14 hours 53 min 36 sec
CPU time	6 days 9 hours 34 min 19 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.75 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12179, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12179, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18417, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=25902, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=25902, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=25902, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=25902, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27092, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27258, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27258, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27258, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27258, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0910f0f0 *** ======= Backtrace: ========= /lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x203591] /lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0x204de8] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x207ecd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0xa7c741] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0xa7c79d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x1aebd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 00198000-002eb000 r-xp 00000000 08:01 278741 /lib/tls/i686/cmov/libc-2.11.1.so 002eb000-002ec000 ---p 00153000 08:01 278741 /lib/tls/i686/cmov/libc-2.11.1.so 002ec000-002ee000 r--p 00153000 08:01 278741 /lib/tls/i686/cmov/libc-2.11.1.so 002ee000-002ef000 rw-p 00155000 08:01 278741 /lib/tls/i686/cmov/libc-2.11.1.so 002ef000-002f2000 rw-p 00000000 00:00 0 00631000-0064c000 r-xp 00000000 08:01 1753102 /lib/ld-2.11.1.so 0064c000-0064d000 r--p 0001a000 08:01 1753102 /lib/ld-2.11.1.so 0064d000-0064e000 rw-p 0001b000 08:01 1753102 /lib/ld-2.11.1.so 0066b000-0066d000 r-xp 00000000 08:01 278745 /lib/tls/i686/cmov/libdl-2.11.1.so 0066d000-0066e000 r--p 00001000 08:01 278745 /lib/tls/i686/cmov/libdl-2.11.1.so 0066e000-0066f000 rw-p 00002000 08:01 278745 /lib/tls/i686/cmov/libdl-2.11.1.so 006ad000-006ae000 r-xp 00000000 00:00 0 [vdso] 006bc000-006d1000 r-xp 00000000 08:01 278756 /lib/tls/i686/cmov/libpthread-2.11.1.so 006d1000-006d2000 r--p 00014000 08:01 278756 /lib/tls/i686/cmov/libpthread-2.11.1.so 006d2000-006d3000 rw-p 00015000 08:01 278756 /lib/tls/i686/cmov/libpthread-2.11.1.so 006d3000-006d5000 rw-p 00000000 00:00 0 009c1000-00aaa000 r-xp 00000000 08:01 606625 /usr/lib/libstdc++.so.6.0.13 00aaa000-00aab000 ---p 000e9000 08:01 606625 /usr/lib/libstdc++.so.6.0.13 00aab000-00aaf000 r--p 000e9000 08:01 606625 /usr/lib/libstdc++.so.6.0.13 00aaf000-00ab0000 rw-p 000ed000 08:01 606625 /usr/lib/libstdc++.so.6.0.13 00ab0000-00ab7000 rw-p 00000000 00:00 0 00bc6000-00be3000 r-xp 00000000 08:01 1753213 /lib/libgcc_s.so.1 00be3000-00be4000 r--p 0001c000 08:01 1753213 /lib/libgcc_s.so.1 00be4000-00be5000 rw-p 0001d000 08:01 1753213 /lib/libgcc_s.so.1 00fa9000-00fcd000 r-xp 00000000 08:01 278746 /lib/tls/i686/cmov/libm-2.11.1.so 00fcd000-00fce000 r--p 00023000 08:01 278746 /lib/tls/i686/cmov/libm-2.11.1.so 00fce000-00fcf000 rw-p 00024000 08:01 278746 /lib/tls/i686/cmov/libm-2.11.1.so 08048000-080e3000 r-xp 00000000 08:01 360480 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 360480 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 090b9000-0911f000 rw-p 00000000 00:00 0 [heap] b7200000-b7221000 rw-p 00000000 00:00 0 b7221000-b7300000 ---p 00000000 00:00 0 b7398000-b7813000 rw-s 00000000 08:01 434568 /var/lib/boinc-client/slots/2/134565 b7813000-b7816000 rw-p 00000000 00:00 0 b782a000-b782b000 rw-p 00000000 00:00 0 b782b000-b782c000 ---p 00000000 00:00 0 b782c000-b782f000 rw-p 00000000 00:00 0 b782f000-b7831000 rw-s 00000000 08:01 434565 /var/lib/boinc-client/slots/2/boinc_mmap_file b7831000-b7833000 rw-p 00000000 00:00 0 bfa16000-bfa87000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x6ad400] [0x6ad422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0x1c2651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0x1c5a82] /lib/tls/i686/cmov/libc.so.6(+0x6149d)[0x1f949d] /lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x203591] /lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0x204de8] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x207ecd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0xa7c741] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0xa7c79d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x1aebd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Oct 2011 07:58:26	987797	13408518	hadcm3n_u1n2_1980_40_007459056_0	259,200	552,852	2.1329
11 Oct 2011 17:53:41	987797	13408518	hadcm3n_u1n2_1980_40_007459056_0	233,280	497,362	2.1320
09 Oct 2011 10:53:35	987797	13408518	hadcm3n_u1n2_1980_40_007459056_0	207,360	441,790	2.1305
08 Oct 2011 17:36:29	987797	13408518	hadcm3n_u1n2_1980_40_007459056_0	181,440	385,769	2.1262
06 Oct 2011 03:22:53	987797	13408518	hadcm3n_u1n2_1980_40_007459056_0	155,520	329,794	2.1206
04 Oct 2011 19:35:36	987797	13408518	hadcm3n_u1n2_1980_40_007459056_0	129,600	273,896	2.1134
01 Oct 2011 10:04:40	987797	13408518	hadcm3n_u1n2_1980_40_007459056_0	103,680	218,449	2.1070
29 Sep 2011 10:37:08	987797	13408518	hadcm3n_u1n2_1980_40_007459056_0	77,760	162,806	2.0937
26 Sep 2011 13:48:07	987797	13408518	hadcm3n_u1n2_1980_40_007459056_0	51,840	106,841	2.0610
24 Sep 2011 02:37:35	987797	13408518	hadcm3n_u1n2_1980_40_007459056_0	25,920	50,935	1.9651