Task 13028061

Name	hadcm3n_t7c8_1940_40_007317078_0
Workunit	7514508
Created	29 Jun 2011, 6:21:43 UTC
Sent	29 Jun 2011, 11:33:50 UTC
Report deadline	28 Sep 2011, 19:01:01 UTC
Received	18 Jul 2011, 13:44:32 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1147464
Run time	6 days 22 hours 28 min 17 sec
CPU time	6 days 15 hours 33 min 15 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.16 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.59</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:39:36 (2041): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:44:17 (2056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:45:58 (2068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:47:40 (2080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:12:30 (2092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:16:32 (2304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:17:44 (2321): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:20:06 (2405): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:21:46 (2421): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:23:28 (2437): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:25:48 (2479): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:27:28 (2496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:09 (2512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:30:52 (2528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:32:32 (2544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x082efae0 *** ======= Backtrace: ========= /lib32/libc.so.6(+0x6b511)[0xf7486511] /lib32/libc.so.6(+0x6ce1b)[0xf7487e1b] /lib32/libc.so.6(cfree+0x6d)[0xf748af8d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7664841] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf766489d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf7431e37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 08:11 1843637 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:11 1843637 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 0829a000-082ff000 rw-p 00000000 00:00 0 [heap] f6e00000-f6e21000 rw-p 00000000 00:00 0 f6e21000-f6f00000 ---p 00000000 00:00 0 f6f9d000-f7418000 rw-s 00000000 08:11 1844222 /var/lib/boinc-client/slots/1/133965 f7418000-f741b000 rw-p 00000000 00:00 0 f741b000-f7571000 r-xp 00000000 08:11 1441798 /lib32/libc-2.13.so f7571000-f7572000 ---p 00156000 08:11 1441798 /lib32/libc-2.13.so f7572000-f7574000 r--p 00156000 08:11 1441798 /lib32/libc-2.13.so f7574000-f7575000 rw-p 00158000 08:11 1441798 /lib32/libc-2.13.so f7575000-f7578000 rw-p 00000000 00:00 0 f7578000-f7592000 r-xp 00000000 08:11 2503463 /usr/lib32/libgcc_s.so.1 f7592000-f7593000 r--p 00019000 08:11 2503463 /usr/lib32/libgcc_s.so.1 f7593000-f7594000 rw-p 0001a000 08:11 2503463 /usr/lib32/libgcc_s.so.1 f7594000-f75b8000 r-xp 00000000 08:11 1441802 /lib32/libm-2.13.so f75b8000-f75b9000 r--p 00023000 08:11 1441802 /lib32/libm-2.13.so f75b9000-f75ba000 rw-p 00024000 08:11 1441802 /lib32/libm-2.13.so f75ba000-f7699000 r-xp 00000000 08:11 2503466 /usr/lib32/libstdc++.so.6.0.14 f7699000-f769d000 r--p 000de000 08:11 2503466 /usr/lib32/libstdc++.so.6.0.14 f769d000-f769e000 rw-p 000e2000 08:11 2503466 /usr/lib32/libstdc++.so.6.0.14 f769e000-f76a5000 rw-p 00000000 00:00 0 f76a5000-f76a7000 r-xp 00000000 08:11 1441801 /lib32/libdl-2.13.so f76a7000-f76a8000 r--p 00001000 08:11 1441801 /lib32/libdl-2.13.so f76a8000-f76a9000 rw-p 00002000 08:11 1441801 /lib32/libdl-2.13.so f76a9000-f76be000 r-xp 00000000 08:11 1441812 /lib32/libpthread-2.13.so f76be000-f76bf000 r--p 00014000 08:11 1441812 /lib32/libpthread-2.13.so f76bf000-f76c0000 rw-p 00015000 08:11 1441812 /lib32/libpthread-2.13.so f76c0000-f76c3000 rw-p 00000000 00:00 0 f76dd000-f76de000 rw-p 00000000 00:00 0 f76de000-f76df000 ---p 00000000 00:00 0 f76df000-f76e2000 rw-p 00000000 00:00 0 f76e2000-f76e4000 rw-s 00000000 08:11 1844131 /var/lib/boinc-client/slots/1/boinc_mmap_file f76e4000-f76e5000 rw-p 00000000 00:00 0 f76e5000-f76e6000 r-xp 00000000 00:00 0 [vdso] f76e6000-f7702000 r-xp 00000000 08:11 1441795 /lib32/ld-2.13.so f7702000-f7703000 r--p 0001b000 08:11 1441795 /lib32/ld-2.13.so f7703000-f7704000 rw-p 0001c000 08:11 1441795 /lib32/ld-2.13.so fff2d000-fff9c000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf76e5400] [0xf76e5425] /lib32/libc.so.6(gsignal+0x51)[0xf7445ea1] /lib32/libc.so.6(abort+0x17e)[0xf74492ce] /lib32/libc.so.6(+0x61277)[0xf747c277] /lib32/libc.so.6(+0x6b511)[0xf7486511] /lib32/libc.so.6(+0x6ce1b)[0xf7487e1b] /lib32/libc.so.6(cfree+0x6d)[0xf748af8d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7664841] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf766489d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf7431e37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 17:24:41	1147464	13028061	hadcm3n_t7c8_1940_40_007317078_0	259,200	574,389	2.2160
25 Jul 2011 17:20:42	1147464	13028061	hadcm3n_t7c8_1940_40_007317078_0	233,280	516,762	2.2152
25 Jul 2011 17:20:42	1147464	13028061	hadcm3n_t7c8_1940_40_007317078_0	207,360	459,283	2.2149
07 Jul 2011 21:09:30	1147464	13028061	hadcm3n_t7c8_1940_40_007317078_0	181,440	401,957	2.2154
07 Jul 2011 15:42:38	1147464	13028061	hadcm3n_t7c8_1940_40_007317078_0	155,520	344,480	2.2150
05 Jul 2011 09:58:31	1147464	13028061	hadcm3n_t7c8_1940_40_007317078_0	129,600	287,113	2.2154
04 Jul 2011 17:22:22	1147464	13028061	hadcm3n_t7c8_1940_40_007317078_0	103,680	229,869	2.2171
03 Jul 2011 07:54:01	1147464	13028061	hadcm3n_t7c8_1940_40_007317078_0	77,760	172,328	2.2162
01 Jul 2011 12:10:02	1147464	13028061	hadcm3n_t7c8_1940_40_007317078_0	51,840	115,010	2.2186
30 Jun 2011 09:40:04	1147464	13028061	hadcm3n_t7c8_1940_40_007317078_0	25,920	57,546	2.2201