Name | hadcm3n_z8h3_1880_40_008249208_3 |
Workunit | 8404332 |
Created | 26 Nov 2012, 23:48:14 UTC |
Sent | 26 Nov 2012, 23:48:29 UTC |
Report deadline | 26 Feb 2013, 7:15:40 UTC |
Received | 9 Mar 2013, 22:15:04 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Aborted by user |
Exit status | 203 (0x000000CB) EXIT_ABORTED_VIA_GUI |
Computer ID | 1172856 |
Run time | 33 days 17 hours 44 min 25 sec |
CPU time | 29 days 20 hours 0 min 50 sec |
Validate state | Invalid |
Credit | 12,441.60 |
Device peak FLOPS | 2.45 GFLOPS |
Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu |
Stderr | <core_client_version>7.0.27</core_client_version> <![CDATA[ <message> aborted by user </message> <stderr_txt> 08:34:34 (6274): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:40:44 (24175): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:53:18 (25861): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:10:27 (18539): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:10:36 (18539): No heartbeat from core client for 30 sec - exiting 07:45:13 (7944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:00:03 (13105): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:11:33 (2738): No heartbeat from core client for 30 sec - exiting 08:11:34 (2738): No heartbeat from core client for 30 sec - exiting 08:11:35 (2738): No heartbeat from core client for 30 sec - exiting 08:11:36 (2738): No heartbeat from core client for 30 sec - exiting 08:12:07 (2738): No heartbeat from core client for 30 sec - exiting 08:12:08 (2738): No heartbeat from core client for 30 sec - exiting 08:12:09 (2738): No heartbeat from core client for 30 sec - exiting 08:12:10 (2738): No heartbeat from core client for 30 sec - exiting 08:12:11 (2738): No heartbeat from core client for 30 sec - exiting 08:12:12 (2738): No heartbeat from core client for 30 sec - exiting 08:12:13 (2738): No heartbeat from core client for 30 sec - exiting 08:12:14 (2738): No heartbeat from core client for 30 sec - exiting 08:12:15 (2738): No heartbeat from core client for 30 sec - exiting 08:12:16 (2738): No heartbeat from core client for 30 sec - exiting 08:12:17 (2738): No heartbeat from core client for 30 sec - exiting 08:12:18 (2738): No heartbeat from core client for 30 sec - exiting 08:12:19 (2738): No heartbeat from core client for 30 sec - exiting 08:12:20 (2738): No heartbeat from core client for 30 sec - exiting 08:12:21 (2738): No heartbeat from core client for 30 sec - exiting 08:12:22 (2738): No heartbeat from core client for 30 sec - exiting 08:12:23 (2738): No heartbeat from core client for 30 sec - exiting 08:12:24 (2738): No heartbeat from core client for 30 sec - exiting 08:12:25 (2738): No heartbeat from core client for 30 sec - exiting 08:12:26 (2738): No heartbeat from core client for 30 sec - exiting 08:12:27 (2738): No heartbeat from core client for 30 sec - exiting 08:12:28 (2738): No heartbeat from core client for 30 sec - exiting 08:12:29 (2738): No heartbeat from core client for 30 sec - exiting 08:12:30 (2738): No heartbeat from core client for 30 sec - exiting 08:12:31 (2738): No heartbeat from core client for 30 sec - exiting 08:12:32 (2738): No heartbeat from core client for 30 sec - exiting 08:12:33 (2738): No heartbeat from core client for 30 sec - exiting 08:12:34 (2738): No heartbeat from core client for 30 sec - exiting 08:12:35 (2738): No heartbeat from core client for 30 sec - exiting 08:12:36 (2738): No heartbeat from core client for 30 sec - exiting 08:12:37 (2738): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:24:23 (2324): No heartbeat from core client for 30 sec - exiting 08:24:25 (2324): No heartbeat from core client for 30 sec - exiting 08:24:26 (2324): No heartbeat from core client for 30 sec - exiting 08:24:27 (2324): No heartbeat from core client for 30 sec - exiting 08:24:28 (2324): No heartbeat from core client for 30 sec - exiting 08:24:29 (2324): No heartbeat from core client for 30 sec - exiting 08:24:30 (2324): No heartbeat from core client for 30 sec - exiting 08:24:31 (2324): No heartbeat from core client for 30 sec - exiting 08:24:32 (2324): No heartbeat from core client for 30 sec - exiting 08:24:35 (2324): No heartbeat from core client for 30 sec - exiting 08:24:36 (2324): No heartbeat from core client for 30 sec - exiting 08:24:38 (2324): No heartbeat from core client for 30 sec - exiting 08:24:39 (2324): No heartbeat from core client for 30 sec - exiting 08:24:40 (2324): No heartbeat from core client for 30 sec - exiting 08:24:41 (2324): No heartbeat from core client for 30 sec - exiting 08:24:42 (2324): No heartbeat from core client for 30 sec - exiting 08:24:43 (2324): No heartbeat from core client for 30 sec - exiting 08:24:44 (2324): No heartbeat from core client for 30 sec - exiting 08:24:45 (2324): No heartbeat from core client for 30 sec - exiting 08:24:46 (2324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:08:57 (11046): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:09:02 (11046): No heartbeat from core client for 30 sec - exiting 09:16:57 (26412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:35:07 (2273): No heartbeat from core client for 30 sec - exiting 09:35:08 (2273): No heartbeat from core client for 30 sec - exiting 09:35:09 (2273): No heartbeat from core client for 30 sec - exiting 09:35:10 (2273): No heartbeat from core client for 30 sec - exiting 09:35:11 (2273): No heartbeat from core client for 30 sec - exiting 09:35:12 (2273): No heartbeat from core client for 30 sec - exiting 09:35:13 (2273): No heartbeat from core client for 30 sec - exiting 09:35:14 (2273): No heartbeat from core client for 30 sec - exiting 09:35:15 (2273): No heartbeat from core client for 30 sec - exiting 09:35:16 (2273): No heartbeat from core client for 30 sec - exiting 09:35:17 (2273): No heartbeat from core client for 30 sec - exiting 09:35:18 (2273): No heartbeat from core client for 30 sec - exiting 09:35:19 (2273): No heartbeat from core client for 30 sec - exiting 09:35:24 (2273): No heartbeat from core client for 30 sec - exiting 09:35:25 (2273): No heartbeat from core client for 30 sec - exiting 09:35:26 (2273): No heartbeat from core client for 30 sec - exiting 09:35:27 (2273): No heartbeat from core client for 30 sec - exiting 09:35:28 (2273): No heartbeat from core client for 30 sec - exiting 09:35:29 (2273): No heartbeat from core client for 30 sec - exiting 09:35:30 (2273): No heartbeat from core client for 30 sec - exiting 09:35:31 (2273): No heartbeat from core client for 30 sec - exiting 09:35:32 (2273): No heartbeat from core client for 30 sec - exiting 09:35:33 (2273): No heartbeat from core client for 30 sec - exiting 09:35:34 (2273): No heartbeat from core client for 30 sec - exiting 09:35:35 (2273): No heartbeat from core client for 30 sec - exiting 09:35:36 (2273): No heartbeat from core client for 30 sec - exiting 09:35:37 (2273): No heartbeat from core client for 30 sec - exiting 09:35:38 (2273): No heartbeat from core client for 30 sec - exiting 09:35:39 (2273): No heartbeat from core client for 30 sec - exiting 09:35:40 (2273): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:37:07 (2194): No heartbeat from core client for 30 sec - exiting 15:37:10 (2194): No heartbeat from core client for 30 sec - exiting 15:37:11 (2194): No heartbeat from core client for 30 sec - exiting 15:37:12 (2194): No heartbeat from core client for 30 sec - exiting 15:37:13 (2194): No heartbeat from core client for 30 sec - exiting 15:37:14 (2194): No heartbeat from core client for 30 sec - exiting 15:37:15 (2194): No heartbeat from core client for 30 sec - exiting 15:37:16 (2194): No heartbeat from core client for 30 sec - exiting 15:37:17 (2194): No heartbeat from core client for 30 sec - exiting 15:37:18 (2194): No heartbeat from core client for 30 sec - exiting 15:37:19 (2194): No heartbeat from core client for 30 sec - exiting 15:37:20 (2194): No heartbeat from core client for 30 sec - exiting 15:37:21 (2194): No heartbeat from core client for 30 sec - exiting 15:37:22 (2194): No heartbeat from core client for 30 sec - exiting 15:37:23 (2194): No heartbeat from core client for 30 sec - exiting 15:37:24 (2194): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:01:35 (2202): No heartbeat from core client for 30 sec - exiting 21:01:36 (2202): No heartbeat from core client for 30 sec - exiting 21:01:37 (2202): No heartbeat from core client for 30 sec - exiting 21:01:38 (2202): No heartbeat from core client for 30 sec - exiting 21:01:39 (2202): No heartbeat from core client for 30 sec - exiting 21:01:40 (2202): No heartbeat from core client for 30 sec - exiting 21:01:41 (2202): No heartbeat from core client for 30 sec - exiting 21:01:42 (2202): No heartbeat from core client for 30 sec - exiting 21:01:43 (2202): No heartbeat from core client for 30 sec - exiting 21:01:44 (2202): No heartbeat from core client for 30 sec - exiting 21:01:45 (2202): No heartbeat from core client for 30 sec - exiting 21:01:46 (2202): No heartbeat from core client for 30 sec - exiting 21:01:47 (2202): No heartbeat from core client for 30 sec - exiting 21:01:48 (2202): No heartbeat from core client for 30 sec - exiting 21:01:49 (2202): No heartbeat from core client for 30 sec - exiting 21:01:50 (2202): No heartbeat from core client for 30 sec - exiting 21:01:51 (2202): No heartbeat from core client for 30 sec - exiting 21:01:52 (2202): No heartbeat from core client for 30 sec - exiting 21:01:53 (2202): No heartbeat from core client for 30 sec - exiting 21:01:54 (2202): No heartbeat from core client for 30 sec - exiting 21:01:55 (2202): No heartbeat from core client for 30 sec - exiting 21:01:56 (2202): No heartbeat from core client for 30 sec - exiting 21:01:57 (2202): No heartbeat from core client for 30 sec - exiting 21:01:58 (2202): No heartbeat from core client for 30 sec - exiting 21:01:59 (2202): No heartbeat from core client for 30 sec - exiting 21:02:00 (2202): No heartbeat from core client for 30 sec - exiting 21:02:01 (2202): No heartbeat from core client for 30 sec - exiting 21:02:02 (2202): No heartbeat from core client for 30 sec - exiting 21:02:03 (2202): No heartbeat from core client for 30 sec - exiting 21:02:04 (2202): No heartbeat from core client for 30 sec - exiting 21:02:05 (2202): No heartbeat from core client for 30 sec - exiting 21:02:06 (2202): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:43:05 (2176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:50:36 (2184): No heartbeat from core client for 30 sec - exiting 09:50:38 (2184): No heartbeat from core client for 30 sec - exiting 09:50:40 (2184): No heartbeat from core client for 30 sec - exiting 09:50:41 (2184): No heartbeat from core client for 30 sec - exiting 09:50:42 (2184): No heartbeat from core client for 30 sec - exiting 09:50:43 (2184): No heartbeat from core client for 30 sec - exiting 09:50:44 (2184): No heartbeat from core client for 30 sec - exiting 09:50:45 (2184): No heartbeat from core client for 30 sec - exiting 09:50:46 (2184): No heartbeat from core client for 30 sec - exiting 09:50:47 (2184): No heartbeat from core client for 30 sec - exiting 09:50:48 (2184): No heartbeat from core client for 30 sec - exiting 09:50:49 (2184): No heartbeat from core client for 30 sec - exiting 09:50:50 (2184): No heartbeat from core client for 30 sec - exiting 09:50:51 (2184): No heartbeat from core client for 30 sec - exiting 09:50:52 (2184): No heartbeat from core client for 30 sec - exiting 09:50:53 (2184): No heartbeat from core client for 30 sec - exiting 09:50:54 (2184): No heartbeat from core client for 30 sec - exiting 09:50:55 (2184): No heartbeat from core client for 30 sec - exiting 09:50:58 (2184): No heartbeat from core client for 30 sec - exiting 09:50:59 (2184): No heartbeat from core client for 30 sec - exiting 09:51:00 (2184): No heartbeat from core client for 30 sec - exiting 09:51:01 (2184): No heartbeat from core client for 30 sec - exiting 09:51:02 (2184): No heartbeat from core client for 30 sec - exiting 09:51:03 (2184): No heartbeat from core client for 30 sec - exiting 09:51:04 (2184): No heartbeat from core client for 30 sec - exiting 09:51:05 (2184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:40:05 (2242): No heartbeat from core client for 30 sec - exiting 09:40:06 (2242): No heartbeat from core client for 30 sec - exiting 09:40:07 (2242): No heartbeat from core client for 30 sec - exiting 09:40:08 (2242): No heartbeat from core client for 30 sec - exiting 09:40:09 (2242): No heartbeat from core client for 30 sec - exiting 09:40:10 (2242): No heartbeat from core client for 30 sec - exiting 09:40:11 (2242): No heartbeat from core client for 30 sec - exiting 09:40:12 (2242): No heartbeat from core client for 30 sec - exiting 09:40:13 (2242): No heartbeat from core client for 30 sec - exiting 09:40:14 (2242): No heartbeat from core client for 30 sec - exiting 09:40:15 (2242): No heartbeat from core client for 30 sec - exiting 09:40:16 (2242): No heartbeat from core client for 30 sec - exiting 09:40:17 (2242): No heartbeat from core client for 30 sec - exiting 09:40:18 (2242): No heartbeat from core client for 30 sec - exiting 09:40:19 (2242): No heartbeat from core client for 30 sec - exiting 09:40:20 (2242): No heartbeat from core client for 30 sec - exiting 09:40:21 (2242): No heartbeat from core client for 30 sec - exiting 09:40:22 (2242): No heartbeat from core client for 30 sec - exiting 09:40:23 (2242): No heartbeat from core client for 30 sec - exiting 09:40:24 (2242): No heartbeat from core client for 30 sec - exiting 09:40:25 (2242): No heartbeat from core client for 30 sec - exiting 09:40:26 (2242): No heartbeat from core client for 30 sec - exiting 09:40:27 (2242): No heartbeat from core client for 30 sec - exiting 09:40:28 (2242): No heartbeat from core client for 30 sec - exiting 09:40:29 (2242): No heartbeat from core client for 30 sec - exiting 09:40:30 (2242): No heartbeat from core client for 30 sec - exiting 09:40:31 (2242): No heartbeat from core client for 30 sec - exiting 09:40:32 (2242): No heartbeat from core client for 30 sec - exiting 09:40:33 (2242): No heartbeat from core client for 30 sec - exiting 09:40:34 (2242): No heartbeat from core client for 30 sec - exiting 09:40:35 (2242): No heartbeat from core client for 30 sec - exiting 09:40:36 (2242): No heartbeat from core client for 30 sec - exiting 09:40:37 (2242): No heartbeat from core client for 30 sec - exiting 09:40:38 (2242): No heartbeat from core client for 30 sec - exiting 09:40:39 (2242): No heartbeat from core client for 30 sec - exiting 09:40:40 (2242): No heartbeat from core client for 30 sec - exiting 09:40:41 (2242): No heartbeat from core client for 30 sec - exiting 09:40:42 (2242): No heartbeat from core client for 30 sec - exiting 09:40:43 (2242): No heartbeat from core client for 30 sec - exiting 09:40:44 (2242): No heartbeat from core client for 30 sec - exiting 09:40:45 (2242): No heartbeat from core client for 30 sec - exiting 09:40:46 (2242): No heartbeat from core client for 30 sec - exiting 09:40:47 (2242): No heartbeat from core client for 30 sec - exiting 09:40:48 (2242): No heartbeat from core client for 30 sec - exiting 09:40:49 (2242): No heartbeat from core client for 30 sec - exiting 09:40:50 (2242): No heartbeat from core client for 30 sec - exiting 09:40:51 (2242): No heartbeat from core client for 30 sec - exiting 09:40:52 (2242): No heartbeat from core client for 30 sec - exiting 09:40:53 (2242): No heartbeat from core client for 30 sec - exiting 09:40:54 (2242): No heartbeat from core client for 30 sec - exiting 09:40:55 (2242): No heartbeat from core client for 30 sec - exiting 09:40:56 (2242): No heartbeat from core client for 30 sec - exiting 09:40:57 (2242): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08998298 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 10:28:49 (6758): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09e0a750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 01:33:26 (5267): No heartbeat from core client for 30 sec - exiting 01:33:28 (5267): No heartbeat from core client for 30 sec - exiting 01:33:29 (5267): No heartbeat from core client for 30 sec - exiting 01:33:30 (5267): No heartbeat from core client for 30 sec - exiting 01:33:31 (5267): No heartbeat from core client for 30 sec - exiting 01:33:32 (5267): No heartbeat from core client for 30 sec - exiting 01:33:33 (5267): No heartbeat from core client for 30 sec - exiting 01:33:34 (5267): No heartbeat from core client for 30 sec - exiting 01:33:35 (5267): No heartbeat from core client for 30 sec - exiting 01:33:36 (5267): No heartbeat from core client for 30 sec - exiting 01:33:37 (5267): No heartbeat from core client for 30 sec - exiting 01:33:38 (5267): No heartbeat from core client for 30 sec - exiting 01:33:39 (5267): No heartbeat from core client for 30 sec - exiting 01:33:40 (5267): No heartbeat from core client for 30 sec - exiting 01:33:41 (5267): No heartbeat from core client for 30 sec - exiting 01:33:42 (5267): No heartbeat from core client for 30 sec - exiting 01:33:43 (5267): No heartbeat from core client for 30 sec - exiting 01:33:44 (5267): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x094ff750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 01:39:29 (2721): No heartbeat from core client for 30 sec - exiting 01:39:30 (2721): No heartbeat from core client for 30 sec - exiting 01:39:31 (2721): No heartbeat from core client for 30 sec - exiting 01:39:32 (2721): No heartbeat from core client for 30 sec - exiting 01:39:33 (2721): No heartbeat from core client for 30 sec - exiting 01:39:34 (2721): No heartbeat from core client for 30 sec - exiting 01:52:11 (2256): No heartbeat from core client for 30 sec - exiting 01:52:13 (2256): No heartbeat from core client for 30 sec - exiting 01:52:48 (2256): No heartbeat from core client for 30 sec - exiting 01:52:49 (2256): No heartbeat from core client for 30 sec - exiting 01:52:50 (2256): No heartbeat from core client for 30 sec - exiting 01:52:51 (2256): No heartbeat from core client for 30 sec - exiting 01:52:52 (2256): No heartbeat from core client for 30 sec - exiting 01:52:53 (2256): No heartbeat from core client for 30 sec - exiting 01:52:54 (2256): No heartbeat from core client for 30 sec - exiting 01:52:55 (2256): No heartbeat from core client for 30 sec - exiting 01:52:56 (2256): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0936f750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 01:52:57 (2256): No heartbeat from core client for 30 sec - exiting 01:52:58 (2256): No heartbeat from core client for 30 sec - exiting 01:52:59 (2256): No heartbeat from core client for 30 sec - exiting 01:53:00 (2256): No heartbeat from core client for 30 sec - exiting 01:53:01 (2256): No heartbeat from core client for 30 sec - exiting 01:53:02 (2256): No heartbeat from core client for 30 sec - exiting 01:53:03 (2256): No heartbeat from core client for 30 sec - exiting 01:53:04 (2256): No heartbeat from core client for 30 sec - exiting 01:53:05 (2256): No heartbeat from core client for 30 sec - exiting 01:53:06 (2256): No heartbeat from core client for 30 sec - exiting 01:53:07 (2256): No heartbeat from core client for 30 sec - exiting 01:53:08 (2256): No heartbeat from core client for 30 sec - exiting 01:53:09 (2256): No heartbeat from core client for 30 sec - exiting 01:53:10 (2256): No heartbeat from core client for 30 sec - exiting 01:53:11 (2256): No heartbeat from core client for 30 sec - exiting 01:53:12 (2256): No heartbeat from core client for 30 sec - exiting 14:20:02 (2256): No heartbeat from core client for 30 sec - exiting 14:20:03 (2256): No heartbeat from core client for 30 sec - exiting 14:20:04 (2256): No heartbeat from core client for 30 sec - exiting 14:20:05 (2256): No heartbeat from core client for 30 sec - exiting 14:20:06 (2256): No heartbeat from core client for 30 sec - exiting 14:20:07 (2256): No heartbeat from core client for 30 sec - exiting 14:20:08 (2256): No heartbeat from core client for 30 sec - exiting 14:20:09 (2256): No heartbeat from core client for 30 sec - exiting 14:20:10 (2256): No heartbeat from core client for 30 sec - exiting 14:20:11 (2256): No heartbeat from core client for 30 sec - exiting 14:20:12 (2256): No heartbeat from core client for 30 sec - exiting 14:20:13 (2256): No heartbeat from core client for 30 sec - exiting 14:20:14 (2256): No heartbeat from core client for 30 sec - exiting 14:40:24 (2256): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0955b750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08821750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08a41750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08fc3750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0990b750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 12:04:43 (1897): No heartbeat from core client for 30 sec - exiting 12:04:45 (1897): No heartbeat from core client for 30 sec - exiting 12:04:46 (1897): No heartbeat from core client for 30 sec - exiting 12:04:47 (1897): No heartbeat from core client for 30 sec - exiting 12:04:48 (1897): No heartbeat from core client for 30 sec - exiting 12:04:50 (1897): No heartbeat from core client for 30 sec - exiting 12:04:51 (1897): No heartbeat from core client for 30 sec - exiting 12:04:52 (1897): No heartbeat from core client for 30 sec - exiting 12:04:53 (1897): No heartbeat from core client for 30 sec - exiting 12:04:54 (1897): No heartbeat from core client for 30 sec - exiting 12:04:56 (1897): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09f4f750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 18:04:31 (2172): No heartbeat from core client for 30 sec - exiting 18:04:35 (2172): No heartbeat from core client for 30 sec - exiting 18:04:36 (2172): No heartbeat from core client for 30 sec - exiting 18:04:37 (2172): No heartbeat from core client for 30 sec - exiting 18:04:38 (2172): No heartbeat from core client for 30 sec - exiting 18:04:39 (2172): No heartbeat from core client for 30 sec - exiting 18:04:40 (2172): No heartbeat from core client for 30 sec - exiting 18:04:41 (2172): No heartbeat from core client for 30 sec - exiting 18:04:42 (2172): No heartbeat from core client for 30 sec - exiting 18:04:43 (2172): No heartbeat from core client for 30 sec - exiting 18:04:44 (2172): No heartbeat from core client for 30 sec - exiting 18:04:45 (2172): No heartbeat from core client for 30 sec - exiting 18:04:46 (2172): No heartbeat from core client for 30 sec - exiting 18:04:47 (2172): No heartbeat from core client for 30 sec - exiting 18:04:48 (2172): No heartbeat from core client for 30 sec - exiting 18:04:49 (2172): No heartbeat from core client for 30 sec - exiting 18:04:50 (2172): No heartbeat from core client for 30 sec - exiting 18:04:51 (2172): No heartbeat from core client for 30 sec - exiting 18:04:52 (2172): No heartbeat from core client for 30 sec - exiting 18:04:53 (2172): No heartbeat from core client for 30 sec - exiting 18:04:54 (2172): No heartbeat from core client for 30 sec - exiting 18:04:55 (2172): No heartbeat from core client for 30 sec - exiting 18:04:56 (2172): No heartbeat from core client for 30 sec - exiting 18:04:57 (2172): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09952750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 18:20:53 (2172): No heartbeat from core client for 30 sec - exiting 18:20:54 (2172): No heartbeat from core client for 30 sec - exiting 18:20:55 (2172): No heartbeat from core client for 30 sec - exiting 18:20:56 (2172): No heartbeat from core client for 30 sec - exiting 18:20:57 (2172): No heartbeat from core client for 30 sec - exiting 18:20:58 (2172): No heartbeat from core client for 30 sec - exiting 18:20:59 (2172): No heartbeat from core client for 30 sec - exiting 18:21:00 (2172): No heartbeat from core client for 30 sec - exiting 18:21:01 (2172): No heartbeat from core client for 30 sec - exiting 18:21:02 (2172): No heartbeat from core client for 30 sec - exiting 18:21:03 (2172): No heartbeat from core client for 30 sec - exiting 18:22:46 (2172): No heartbeat from core client for 30 sec - exiting 18:22:47 (2172): No heartbeat from core client for 30 sec - exiting 18:22:48 (2172): No heartbeat from core client for 30 sec - exiting 18:22:49 (2172): No heartbeat from core client for 30 sec - exiting 18:22:50 (2172): No heartbeat from core client for 30 sec - exiting 18:22:51 (2172): No heartbeat from core client for 30 sec - exiting 18:22:52 (2172): No heartbeat from core client for 30 sec - exiting 18:22:53 (2172): No heartbeat from core client for 30 sec - exiting 18:22:54 (2172): No heartbeat from core client for 30 sec - exiting 18:22:55 (2172): No heartbeat from core client for 30 sec - exiting 18:22:56 (2172): No heartbeat from core client for 30 sec - exiting 18:23:33 (2172): No heartbeat from core client for 30 sec - exiting 18:23:34 (2172): No heartbeat from core client for 30 sec - exiting 18:23:35 (2172): No heartbeat from core client for 30 sec - exiting 18:23:36 (2172): No heartbeat from core client for 30 sec - exiting 18:23:37 (2172): No heartbeat from core client for 30 sec - exiting 18:23:38 (2172): No heartbeat from core client for 30 sec - exiting 18:23:39 (2172): No heartbeat from core client for 30 sec - exiting 18:23:40 (2172): No heartbeat from core client for 30 sec - exiting 18:23:41 (2172): No heartbeat from core client for 30 sec - exiting 18:23:42 (2172): No heartbeat from core client for 30 sec - exiting 18:23:43 (2172): No heartbeat from core client for 30 sec - exiting 18:24:51 (2172): No heartbeat from core client for 30 sec - exiting 18:24:52 (2172): No heartbeat from core client for 30 sec - exiting 18:24:53 (2172): No heartbeat from core client for 30 sec - exiting 18:24:54 (2172): No heartbeat from core client for 30 sec - exiting 18:24:55 (2172): No heartbeat from core client for 30 sec - exiting 18:24:56 (2172): No heartbeat from core client for 30 sec - exiting 18:24:57 (2172): No heartbeat from core client for 30 sec - exiting 18:24:58 (2172): No heartbeat from core client for 30 sec - exiting 18:24:59 (2172): No heartbeat from core client for 30 sec - exiting 18:25:00 (2172): No heartbeat from core client for 30 sec - exiting 18:25:01 (2172): No heartbeat from core client for 30 sec - exiting 18:25:38 (2172): No heartbeat from core client for 30 sec - exiting 18:25:39 (2172): No heartbeat from core client for 30 sec - exiting 18:25:40 (2172): No heartbeat from core client for 30 sec - exiting 18:25:41 (2172): No heartbeat from core client for 30 sec - exiting 18:25:42 (2172): No heartbeat from core client for 30 sec - exiting 18:25:43 (2172): No heartbeat from core client for 30 sec - exiting 18:25:44 (2172): No heartbeat from core client for 30 sec - exiting 18:25:45 (2172): No heartbeat from core client for 30 sec - exiting 18:25:46 (2172): No heartbeat from core client for 30 sec - exiting 18:25:47 (2172): No heartbeat from core client for 30 sec - exiting 18:25:48 (2172): No heartbeat from core client for 30 sec - exiting 18:27:36 (2172): No heartbeat from core client for 30 sec - exiting 18:27:37 (2172): No heartbeat from core client for 30 sec - exiting 18:27:38 (2172): No heartbeat from core client for 30 sec - exiting 18:27:39 (2172): No heartbeat from core client for 30 sec - exiting 18:27:40 (2172): No heartbeat from core client for 30 sec - exiting 18:27:41 (2172): No heartbeat from core client for 30 sec - exiting 18:27:42 (2172): No heartbeat from core client for 30 sec - exiting 18:27:43 (2172): No heartbeat from core client for 30 sec - exiting 18:27:44 (2172): No heartbeat from core client for 30 sec - exiting 18:27:45 (2172): No heartbeat from core client for 30 sec - exiting 18:27:46 (2172): No heartbeat from core client for 30 sec - exiting 18:28:23 (2172): No heartbeat from core client for 30 sec - exiting 18:28:24 (2172): No heartbeat from core client for 30 sec - exiting 18:28:25 (2172): No heartbeat from core client for 30 sec - exiting 18:28:26 (2172): No heartbeat from core client for 30 sec - exiting 18:28:27 (2172): No heartbeat from core client for 30 sec - exiting 18:28:28 (2172): No heartbeat from core client for 30 sec - exiting 18:28:29 (2172): No heartbeat from core client for 30 sec - exiting 18:28:30 (2172): No heartbeat from core client for 30 sec - exiting 18:28:31 (2172): No heartbeat from core client for 30 sec - exiting 18:28:32 (2172): No heartbeat from core client for 30 sec - exiting 18:28:33 (2172): No heartbeat from core client for 30 sec - exiting 18:30:01 (2172): No heartbeat from core client for 30 sec - exiting 18:30:02 (2172): No heartbeat from core client for 30 sec - exiting 18:30:03 (2172): No heartbeat from core client for 30 sec - exiting 18:30:04 (2172): No heartbeat from core client for 30 sec - exiting 18:30:05 (2172): No heartbeat from core client for 30 sec - exiting 18:30:06 (2172): No heartbeat from core client for 30 sec - exiting 18:30:07 (2172): No heartbeat from core client for 30 sec - exiting 18:30:08 (2172): No heartbeat from core client for 30 sec - exiting 18:30:09 (2172): No heartbeat from core client for 30 sec - exiting 18:30:10 (2172): No heartbeat from core client for 30 sec - exiting 18:30:11 (2172): No heartbeat from core client for 30 sec - exiting 18:31:29 (2172): No heartbeat from core client for 30 sec - exiting 18:31:30 (2172): No heartbeat from core client for 30 sec - exiting 18:31:31 (2172): No heartbeat from core client for 30 sec - exiting 18:31:32 (2172): No heartbeat from core client for 30 sec - exiting 18:31:33 (2172): No heartbeat from core client for 30 sec - exiting 18:31:34 (2172): No heartbeat from core client for 30 sec - exiting 18:31:35 (2172): No heartbeat from core client for 30 sec - exiting 18:31:36 (2172): No heartbeat from core client for 30 sec - exiting 18:31:37 (2172): No heartbeat from core client for 30 sec - exiting 18:31:38 (2172): No heartbeat from core client for 30 sec - exiting 18:31:39 (2172): No heartbeat from core client for 30 sec - exiting 18:32:16 (2172): No heartbeat from core client for 30 sec - exiting 18:32:17 (2172): No heartbeat from core client for 30 sec - exiting 18:32:18 (2172): No heartbeat from core client for 30 sec - exiting 18:32:19 (2172): No heartbeat from core client for 30 sec - exiting 18:32:20 (2172): No heartbeat from core client for 30 sec - exiting 18:32:21 (2172): No heartbeat from core client for 30 sec - exiting 18:32:22 (2172): No heartbeat from core client for 30 sec - exiting 18:32:23 (2172): No heartbeat from core client for 30 sec - exiting 18:32:24 (2172): No heartbeat from core client for 30 sec - exiting 18:32:25 (2172): No heartbeat from core client for 30 sec - exiting 18:32:26 (2172): No heartbeat from core client for 30 sec - exiting 18:50:40 (2166): No heartbeat from core client for 30 sec - exiting 18:50:42 (2166): No heartbeat from core client for 30 sec - exiting 18:50:43 (2166): No heartbeat from core client for 30 sec - exiting 18:50:44 (2166): No heartbeat from core client for 30 sec - exiting 18:50:45 (2166): No heartbeat from core client for 30 sec - exiting 18:50:46 (2166): No heartbeat from core client for 30 sec - exiting 18:50:47 (2166): No heartbeat from core client for 30 sec - exiting 18:50:51 (2166): No heartbeat from core client for 30 sec - exiting 18:50:52 (2166): No heartbeat from core client for 30 sec - exiting 18:50:53 (2166): No heartbeat from core client for 30 sec - exiting 18:50:54 (2166): No heartbeat from core client for 30 sec - exiting 18:50:55 (2166): No heartbeat from core client for 30 sec - exiting 18:50:56 (2166): No heartbeat from core client for 30 sec - exiting 18:50:57 (2166): No heartbeat from core client for 30 sec - exiting 18:50:58 (2166): No heartbeat from core client for 30 sec - exiting 18:50:59 (2166): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09ba7750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09052750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08f9b750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08358750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0973f750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x083e4750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 17:52:16 (2179): No heartbeat from core client for 30 sec - exiting 17:52:21 (2179): No heartbeat from core client for 30 sec - exiting 17:52:22 (2179): No heartbeat from core client for 30 sec - exiting 17:52:23 (2179): No heartbeat from core client for 30 sec - exiting 17:52:24 (2179): No heartbeat from core client for 30 sec - exiting 17:52:25 (2179): No heartbeat from core client for 30 sec - exiting 17:52:26 (2179): No heartbeat from core client for 30 sec - exiting 17:52:27 (2179): No heartbeat from core client for 30 sec - exiting 17:52:28 (2179): No heartbeat from core client for 30 sec - exiting 17:52:29 (2179): No heartbeat from core client for 30 sec - exiting 17:52:30 (2179): No heartbeat from core client for 30 sec - exiting 17:52:31 (2179): No heartbeat from core client for 30 sec - exiting 17:52:32 (2179): No heartbeat from core client for 30 sec - exiting 17:52:33 (2179): No heartbeat from core client for 30 sec - exiting 17:52:34 (2179): No heartbeat from core client for 30 sec - exiting 17:52:35 (2179): No heartbeat from core client for 30 sec - exiting 17:52:36 (2179): No heartbeat from core client for 30 sec - exiting 17:52:37 (2179): No heartbeat from core client for 30 sec - exiting 17:52:38 (2179): No heartbeat from core client for 30 sec - exiting 17:52:39 (2179): No heartbeat from core client for 30 sec - exiting 17:52:40 (2179): No heartbeat from core client for 30 sec - exiting 17:52:41 (2179): No heartbeat from core client for 30 sec - exiting 17:52:42 (2179): No heartbeat from core client for 30 sec - exiting 17:52:43 (2179): No heartbeat from core client for 30 sec - exiting 17:52:44 (2179): No heartbeat from core client for 30 sec - exiting 17:52:45 (2179): No heartbeat from core client for 30 sec - exiting 17:52:46 (2179): No heartbeat from core client for 30 sec - exiting 17:52:47 (2179): No heartbeat from core client for 30 sec - exiting 17:52:48 (2179): No heartbeat from core client for 30 sec - exiting 17:52:49 (2179): No heartbeat from core client for 30 sec - exiting 17:52:50 (2179): No heartbeat from core client for 30 sec - exiting 17:52:51 (2179): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0a18a750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 17:52:52 (2179): No heartbeat from core client for 30 sec - exiting 17:52:53 (2179): No heartbeat from core client for 30 sec - exiting 17:52:54 (2179): No heartbeat from core client for 30 sec - exiting 17:52:55 (2179): No heartbeat from core client for 30 sec - exiting 17:52:56 (2179): No heartbeat from core client for 30 sec - exiting 10:58:00 (2200): No heartbeat from core client for 30 sec - exiting 10:58:07 (2200): No heartbeat from core client for 30 sec - exiting 10:58:08 (2200): No heartbeat from core client for 30 sec - exiting 10:58:09 (2200): No heartbeat from core client for 30 sec - exiting 10:58:10 (2200): No heartbeat from core client for 30 sec - exiting 10:58:11 (2200): No heartbeat from core client for 30 sec - exiting 10:58:12 (2200): No heartbeat from core client for 30 sec - exiting 10:58:13 (2200): No heartbeat from core client for 30 sec - exiting 10:58:14 (2200): No heartbeat from core client for 30 sec - exiting 10:58:15 (2200): No heartbeat from core client for 30 sec - exiting 10:58:16 (2200): No heartbeat from core client for 30 sec - exiting 10:58:17 (2200): No heartbeat from core client for 30 sec - exiting 10:58:19 (2200): No heartbeat from core client for 30 sec - exiting 10:58:20 (2200): No heartbeat from core client for 30 sec - exiting 10:58:21 (2200): No heartbeat from core client for 30 sec - exiting 10:58:22 (2200): No heartbeat from core client for 30 sec - exiting 10:58:23 (2200): No heartbeat from core client for 30 sec - exiting 10:58:24 (2200): No heartbeat from core client for 30 sec - exiting 10:58:25 (2200): No heartbeat from core client for 30 sec - exiting 10:58:26 (2200): No heartbeat from core client for 30 sec - exiting 10:58:27 (2200): No heartbeat from core client for 30 sec - exiting 10:58:28 (2200): No heartbeat from core client for 30 sec - exiting 10:58:29 (2200): No heartbeat from core client for 30 sec - exiting 10:58:30 (2200): No heartbeat from core client for 30 sec - exiting 10:58:31 (2200): No heartbeat from core client for 30 sec - exiting 10:58:32 (2200): No heartbeat from core client for 30 sec - exiting 10:58:33 (2200): No heartbeat from core client for 30 sec - exiting 10:58:34 (2200): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09447750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 10:58:35 (2200): No heartbeat from core client for 30 sec - exiting 10:58:36 (2200): No heartbeat from core client for 30 sec - exiting 10:58:37 (2200): No heartbeat from core client for 30 sec - exiting 10:58:38 (2200): No heartbeat from core client for 30 sec - exiting 10:58:39 (2200): No heartbeat from core client for 30 sec - exiting 10:58:40 (2200): No heartbeat from core client for 30 sec - exiting 10:58:41 (2200): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08372750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 17:37:23 (2192): No heartbeat from core client for 30 sec - exiting 17:37:26 (2192): No heartbeat from core client for 30 sec - exiting 17:37:27 (2192): No heartbeat from core client for 30 sec - exiting 17:37:28 (2192): No heartbeat from core client for 30 sec - exiting 17:37:29 (2192): No heartbeat from core client for 30 sec - exiting 17:37:31 (2192): No heartbeat from core client for 30 sec - exiting 17:37:32 (2192): No heartbeat from core client for 30 sec - exiting 17:37:33 (2192): No heartbeat from core client for 30 sec - exiting 17:37:34 (2192): No heartbeat from core client for 30 sec - exiting 17:37:35 (2192): No heartbeat from core client for 30 sec - exiting 17:37:36 (2192): No heartbeat from core client for 30 sec - exiting 17:37:37 (2192): No heartbeat from core client for 30 sec - exiting 17:37:38 (2192): No heartbeat from core client for 30 sec - exiting 17:37:39 (2192): No heartbeat from core client for 30 sec - exiting 17:37:40 (2192): No heartbeat from core client for 30 sec - exiting 17:37:41 (2192): No heartbeat from core client for 30 sec - exiting 17:37:42 (2192): No heartbeat from core client for 30 sec - exiting 17:37:43 (2192): No heartbeat from core client for 30 sec - exiting 17:37:44 (2192): No heartbeat from core client for 30 sec - exiting 17:37:45 (2192): No heartbeat from core client for 30 sec - exiting 17:37:46 (2192): No heartbeat from core client for 30 sec - exiting 17:37:47 (2192): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09a8d750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 07:30:09 (2192): No heartbeat from core client for 30 sec - exiting 07:30:42 (2192): No heartbeat from core client for 30 sec - exiting 07:30:43 (2192): No heartbeat from core client for 30 sec - exiting 07:30:44 (2192): No heartbeat from core client for 30 sec - exiting 07:30:45 (2192): No heartbeat from core client for 30 sec - exiting 07:30:46 (2192): No heartbeat from core client for 30 sec - exiting 07:30:47 (2192): No heartbeat from core client for 30 sec - exiting 07:30:48 (2192): No heartbeat from core client for 30 sec - exiting 07:30:49 (2192): No heartbeat from core client for 30 sec - exiting 07:59:01 (2192): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0a05b750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 07:47:49 (11261): No heartbeat from core client for 30 sec - exiting 07:47:51 (11261): No heartbeat from core client for 30 sec - exiting 07:47:52 (11261): No heartbeat from core client for 30 sec - exiting 07:47:53 (11261): No heartbeat from core client for 30 sec - exiting 07:47:54 (11261): No heartbeat from core client for 30 sec - exiting 07:47:55 (11261): No heartbeat from core client for 30 sec - exiting 07:47:56 (11261): No heartbeat from core client for 30 sec - exiting 07:47:57 (11261): No heartbeat from core client for 30 sec - exiting 07:19:48 (11261): No heartbeat from core client for 30 sec - exiting 07:19:49 (11261): No heartbeat from core client for 30 sec - exiting 07:19:50 (11261): No heartbeat from core client for 30 sec - exiting 07:19:53 (11261): No heartbeat from core client for 30 sec - exiting 07:19:54 (11261): No heartbeat from core client for 30 sec - exiting 07:19:55 (11261): No heartbeat from core client for 30 sec - exiting 07:19:56 (11261): No heartbeat from core client for 30 sec - exiting 07:19:57 (11261): No heartbeat from core client for 30 sec - exiting 07:19:58 (11261): No heartbeat from core client for 30 sec - exiting 07:19:59 (11261): No heartbeat from core client for 30 sec - exiting 07:20:00 (11261): No heartbeat from core client for 30 sec - exiting 07:20:01 (11261): No heartbeat from core client for 30 sec - exiting 07:20:02 (11261): No heartbeat from core client for 30 sec - exiting 07:20:03 (11261): No heartbeat from core client for 30 sec - exiting 07:20:04 (11261): No heartbeat from core client for 30 sec - exiting 07:20:05 (11261): No heartbeat from core client for 30 sec - exiting 07:20:06 (11261): No heartbeat from core client for 30 sec - exiting 07:20:07 (11261): No heartbeat from core client for 30 sec - exiting 07:20:08 (11261): No heartbeat from core client for 30 sec - exiting 07:20:09 (11261): No heartbeat from core client for 30 sec - exiting 07:20:10 (11261): No heartbeat from core client for 30 sec - exiting 07:20:11 (11261): No heartbeat from core client for 30 sec - exiting 07:20:12 (11261): No heartbeat from core client for 30 sec - exiting 07:20:13 (11261): No heartbeat from core client for 30 sec - exiting 07:20:14 (11261): No heartbeat from core client for 30 sec - exiting 07:20:15 (11261): No heartbeat from core client for 30 sec - exiting 07:20:16 (11261): No heartbeat from core client for 30 sec - exiting 07:20:17 (11261): No heartbeat from core client for 30 sec - exiting 07:20:18 (11261): No heartbeat from core client for 30 sec - exiting 07:20:19 (11261): No heartbeat from core client for 30 sec - exiting 07:20:20 (11261): No heartbeat from core client for 30 sec - exiting 07:20:21 (11261): No heartbeat from core client for 30 sec - exiting 07:20:22 (11261): No heartbeat from core client for 30 sec - exiting 07:20:23 (11261): No heartbeat from core client for 30 sec - exiting 07:20:24 (11261): No heartbeat from core client for 30 sec - exiting 07:20:25 (11261): No heartbeat from core client for 30 sec - exiting 07:20:26 (11261): No heartbeat from core client for 30 sec - exiting 07:20:27 (11261): No heartbeat from core client for 30 sec - exiting 08:41:41 (2340): No heartbeat from core client for 30 sec - exiting 08:41:45 (2340): No heartbeat from core client for 30 sec - exiting 08:41:46 (2340): No heartbeat from core client for 30 sec - exiting 08:41:47 (2340): No heartbeat from core client for 30 sec - exiting 08:41:48 (2340): No heartbeat from core client for 30 sec - exiting 08:41:49 (2340): No heartbeat from core client for 30 sec - exiting 08:41:50 (2340): No heartbeat from core client for 30 sec - exiting 08:41:51 (2340): No heartbeat from core client for 30 sec - exiting 08:41:52 (2340): No heartbeat from core client for 30 sec - exiting 08:41:53 (2340): No heartbeat from core client for 30 sec - exiting 08:41:54 (2340): No heartbeat from core client for 30 sec - exiting 08:41:55 (2340): No heartbeat from core client for 30 sec - exiting 08:41:56 (2340): No heartbeat from core client for 30 sec - exiting 08:41:57 (2340): No heartbeat from core client for 30 sec - exiting 08:41:58 (2340): No heartbeat from core client for 30 sec - exiting 08:41:59 (2340): No heartbeat from core client for 30 sec - exiting 08:42:00 (2340): No heartbeat from core client for 30 sec - exiting 08:42:01 (2340): No heartbeat from core client for 30 sec - exiting 08:42:02 (2340): No heartbeat from core client for 30 sec - exiting 08:42:03 (2340): No heartbeat from core client for 30 sec - exiting 08:42:04 (2340): No heartbeat from core client for 30 sec - exiting 08:42:05 (2340): No heartbeat from core client for 30 sec - exiting 08:42:06 (2340): No heartbeat from core client for 30 sec - exiting 08:42:07 (2340): No heartbeat from core client for 30 sec - exiting 08:42:08 (2340): No heartbeat from core client for 30 sec - exiting 08:42:09 (2340): No heartbeat from core client for 30 sec - exiting 08:42:10 (2340): No heartbeat from core client for 30 sec - exiting 08:42:11 (2340): No heartbeat from core client for 30 sec - exiting 08:42:12 (2340): No heartbeat from core client for 30 sec - exiting 08:42:13 (2340): No heartbeat from core client for 30 sec - exiting 08:42:14 (2340): No heartbeat from core client for 30 sec - exiting 08:42:15 (2340): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x089c0750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 08:42:16 (2340): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0826b750 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 15:38:19 (9595): No heartbeat from core client for 30 sec - exiting 15:38:32 (9595): No heartbeat from core client for 30 sec - exiting 15:38:33 (9595): No heartbeat from core client for 30 sec - exiting 15:38:34 (9595): No heartbeat from core client for 30 sec - exiting 15:38:35 (9595): No heartbeat from core client for 30 sec - exiting 15:38:36 (9595): No heartbeat from core client for 30 sec - exiting 15:38:37 (9595): No heartbeat from core client for 30 sec - exiting 15:38:38 (9595): No heartbeat from core client for 30 sec - exiting 15:38:39 (9595): No heartbeat from core client for 30 sec - exiting 15:38:40 (9595): No heartbeat from core client for 30 sec - exiting 15:38:41 (9595): No heartbeat from core client for 30 sec - exiting 15:38:42 (9595): No heartbeat from core client for 30 sec - exiting 15:38:43 (9595): No heartbeat from core client for 30 sec - exiting 15:38:44 (9595): No heartbeat from core client for 30 sec - exiting 15:38:45 (9595): No heartbeat from core client for 30 sec - exiting 15:38:46 (9595): No heartbeat from core client for 30 sec - exiting 15:38:47 (9595): No heartbeat from core client for 30 sec - exiting 15:38:48 (9595): No heartbeat from core client for 30 sec - exiting 15:38:49 (9595): No heartbeat from core client for 30 sec - exiting </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
18 Jan 2013 16:56:56 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 1,036,800 | 2,576,563 | 2.4851 |
17 Jan 2013 23:22:15 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 1,010,880 | 2,507,556 | 2.4806 |
16 Jan 2013 23:12:32 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 984,960 | 2,436,252 | 2.4735 |
16 Jan 2013 07:22:15 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 959,040 | 2,371,157 | 2.4724 |
15 Jan 2013 02:07:03 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 933,120 | 2,304,255 | 2.4694 |
14 Jan 2013 03:32:27 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 907,200 | 2,236,576 | 2.4654 |
12 Jan 2013 05:48:47 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 881,280 | 2,168,824 | 2.4610 |
10 Jan 2013 18:34:59 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 855,360 | 2,098,946 | 2.4539 |
09 Jan 2013 23:16:56 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 829,440 | 2,029,007 | 2.4462 |
09 Jan 2013 01:13:26 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 803,520 | 1,960,181 | 2.4395 |
22 Dec 2012 00:07:22 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 777,600 | 1,891,453 | 2.4324 |
21 Dec 2012 00:07:55 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 751,680 | 1,822,416 | 2.4245 |
20 Dec 2012 02:34:56 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 725,760 | 1,754,447 | 2.4174 |
19 Dec 2012 06:56:37 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 699,840 | 1,688,390 | 2.4125 |
18 Dec 2012 12:10:07 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 673,920 | 1,627,053 | 2.4143 |
17 Dec 2012 18:57:12 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 648,000 | 1,568,677 | 2.4208 |
16 Dec 2012 23:02:45 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 622,080 | 1,499,357 | 2.4102 |
16 Dec 2012 02:03:12 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 596,160 | 1,429,015 | 2.3970 |
15 Dec 2012 08:03:27 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 570,240 | 1,367,468 | 2.3981 |
14 Dec 2012 13:26:17 | 1172856 | 15463953 | hadcm3n_z8h3_1880_40_008249208_3 | 544,320 | 1,303,830 | 2.3953 |
©2024 cpdn.org