Task 14042547

Name	hadcm3n_yje4_1940_40_007743649_3
Workunit	7898758
Created	1 Feb 2012, 14:12:07 UTC
Sent	1 Feb 2012, 14:12:55 UTC
Report deadline	2 May 2012, 21:40:06 UTC
Received	26 Feb 2012, 5:00:45 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1189010
Run time	15 days 17 hours 10 min 18 sec
CPU time	15 days 13 hours 38 min 33 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.14 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> 13:36:42 (5684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:39:20 (15362): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:41:51 (15396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:57:52 (15426): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:14:08 (23865): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf778d400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf74dfce7] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf7737400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf7489ce7] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... 19:11:07 (2274): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:12:53 (8841): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:56:15 (8934): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:56:56 (17669): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:16:18 (17689): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:33:55 (2252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:29:58 (6592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:11 (6630): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:17:22 (6663): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:18:54 (6699): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:20:50 (6731): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:23:17 (6763): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:25:44 (6795): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:27:40 (6827): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:29:29 (6860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:02 (6892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 03:42:28 (6930): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 03:45:32 (6962): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:47:54 (6994): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:50:11 (7025): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:52:17 (7057): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:48:37 (7089): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:50:16 (11474): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:54:14 (11522): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:13:59 (11899): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:15:16 (23441): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:17:42 (23492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:18 (23528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:40:20 (23560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:42:10 (1039): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:09:57 (1078): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15705, iMonCtr=1 Model crash detected, will try to restart... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (6 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf770f400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8051147] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf7461ce7] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... 02:57:42 (2277): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:28:12 (29051): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:36 (29089): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:32:47 (29122): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:34:45 (29155): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:36:36 (29190): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:38:37 (29221): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:06 (29254): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:43:42 (29286): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:39 (29318): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:47:23 (29353): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:22:10 (29385): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:37:30 (8273): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:18 (8306): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:42:44 (8338): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:44:06 (8370): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:33 (8402): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:26 (8434): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:23:03 (8471): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:19 (8499): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:21:55 (8537): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:24:26 (8569): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:11:11 (16288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:19:17 (18071): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:14:19 (18111): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:16:43 (4996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:37 (5028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:28:53 (5064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:30:05 (5096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:41:14 (5129): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 02:42:25 (5187): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 02:43:57 (5219): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 03:03:56 (5251): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:47:28 (5284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:48:36 (5321): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:39 (5353): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:10:50 (5385): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:54:29 (5418): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:56:35 (5461): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:59:18 (5493): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:06:20 (5525): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:07:55 (5558): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:38 (5590): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:32:20 (5626): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:43:10 (5659): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:51:58 (5691): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08e4fb70 *** ======= Backtrace: ========= /lib32/libc.so.6(+0x6c421)[0xf753a421] /lib32/libc.so.6(+0x6dcb0)[0xf753bcb0] /lib32/libc.so.6(cfree+0x6d)[0xf753ed5d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf77157b1] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf771580d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf74e4ce7] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 08:01 26740438 /usr/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 26740438 /usr/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 08dfa000-08e5f000 rw-p 00000000 00:00 0 [heap] f6f00000-f6f21000 rw-p 00000000 00:00 0 f6f21000-f7000000 ---p 00000000 00:00 0 f7050000-f74cb000 rw-s 00000000 08:01 27787434 /usr/BOINC/slots/1/137210 f74cb000-f74ce000 rw-p 00000000 00:00 0 f74ce000-f7622000 r-xp 00000000 08:01 6291462 /lib32/libc-2.12.1.so f7622000-f7623000 ---p 00154000 08:01 6291462 /lib32/libc-2.12.1.so f7623000-f7625000 r--p 00154000 08:01 6291462 /lib32/libc-2.12.1.so f7625000-f7626000 rw-p 00156000 08:01 6291462 /lib32/libc-2.12.1.so f7626000-f7629000 rw-p 00000000 00:00 0 f7629000-f7643000 r-xp 00000000 08:01 23074874 /usr/lib32/libgcc_s.so.1 f7643000-f7644000 r--p 00019000 08:01 23074874 /usr/lib32/libgcc_s.so.1 f7644000-f7645000 rw-p 0001a000 08:01 23074874 /usr/lib32/libgcc_s.so.1 f7645000-f7669000 r-xp 00000000 08:01 6291469 /lib32/libm-2.12.1.so f7669000-f766a000 r--p 00023000 08:01 6291469 /lib32/libm-2.12.1.so f766a000-f766b000 rw-p 00024000 08:01 6291469 /lib32/libm-2.12.1.so f766b000-f774a000 r-xp 00000000 08:01 23074877 /usr/lib32/libstdc++.so.6.0.14 f774a000-f774e000 r--p 000de000 08:01 23074877 /usr/lib32/libstdc++.so.6.0.14 f774e000-f774f000 rw-p 000e2000 08:01 23074877 /usr/lib32/libstdc++.so.6.0.14 f774f000-f7756000 rw-p 00000000 00:00 0 f7756000-f7758000 r-xp 00000000 08:01 6291478 /lib32/libdl-2.12.1.so f7758000-f7759000 r--p 00001000 08:01 6291478 /lib32/libdl-2.12.1.so f7759000-f775a000 rw-p 00002000 08:01 6291478 /lib32/libdl-2.12.1.so f775a000-f775b000 rw-p 00000000 00:00 0 f775b000-f7770000 r-xp 00000000 08:01 6291464 /lib32/libpthread-2.12.1.so f7770000-f7771000 r--p 00014000 08:01 6291464 /lib32/libpthread-2.12.1.so f7771000-f7772000 rw-p 00015000 08:01 6291464 /lib32/libpthread-2.12.1.so f7772000-f7774000 rw-p 00000000 00:00 0 f7789000-f778a000 rw-p 00000000 00:00 0 f778a000-f778b000 ---p 00000000 00:00 0 f778b000-f778e000 rw-p 00000000 00:00 0 f778e000-f7790000 rw-s 00000000 08:01 27787398 /usr/BOINC/slots/1/boinc_mmap_file f7790000-f7792000 rw-p 00000000 00:00 0 f7792000-f7793000 r-xp 00000000 00:00 0 [vdso] f7793000-f77af000 r-xp 00000000 08:01 6291470 /lib32/ld-2.12.1.so f77af000-f77b0000 r--p 0001b000 08:01 6291470 /lib32/ld-2.12.1.so f77b0000-f77b1000 rw-p 0001c000 08:01 6291470 /lib32/ld-2.12.1.so ffb6c000-ffbdc000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf7792400] [0xf7792430] /lib32/libc.so.6(gsignal+0x51)[0xf74f8a01] /lib32/libc.so.6(abort+0x182)[0xf74fbe42] /lib32/libc.so.6(+0x62345)[0xf7530345] /lib32/libc.so.6(+0x6c421)[0xf753a421] /lib32/libc.so.6(+0x6dcb0)[0xf753bcb0] /lib32/libc.so.6(cfree+0x6d)[0xf753ed5d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf77157b1] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf771580d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf74e4ce7] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Feb 2012 04:02:25	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	777,600	1,345,109	1.7298
25 Feb 2012 15:27:57	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	751,680	1,300,471	1.7301
25 Feb 2012 02:49:09	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	725,760	1,256,534	1.7313
24 Feb 2012 13:54:27	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	699,840	1,211,508	1.7311
24 Feb 2012 01:51:00	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	673,920	1,166,235	1.7305
23 Feb 2012 12:39:19	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	648,000	1,121,318	1.7304
23 Feb 2012 00:14:33	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	622,080	1,076,084	1.7298
22 Feb 2012 10:50:38	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	596,160	1,031,000	1.7294
21 Feb 2012 22:04:26	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	570,240	986,725	1.7304
21 Feb 2012 09:22:50	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	544,320	941,408	1.7295
20 Feb 2012 20:30:13	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	518,400	895,615	1.7277
20 Feb 2012 07:56:27	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	492,480	851,101	1.7282
19 Feb 2012 19:21:39	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	466,560	805,955	1.7274
19 Feb 2012 06:50:09	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	440,640	760,991	1.7270
18 Feb 2012 17:55:27	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	414,720	715,489	1.7252
18 Feb 2012 05:22:14	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	388,800	670,449	1.7244
17 Feb 2012 16:34:13	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	362,880	625,688	1.7242
17 Feb 2012 03:55:14	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	336,960	580,769	1.7236
15 Feb 2012 01:16:16	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	311,040	535,467	1.7215
14 Feb 2012 12:20:26	1189010	14042547	hadcm3n_yje4_1940_40_007743649_3	285,120	490,487	1.7203