climateprediction.net home page
Task 15813415

Task 15813415

Name hadcm3n_o1j4_1940_40_008381996_0
Workunit 8532855
Created 1 Jun 2013, 3:16:45 UTC
Sent 13 Jun 2013, 22:36:03 UTC
Report deadline 13 Sep 2013, 6:03:14 UTC
Received 28 Dec 2013, 21:56:01 UTC
Server state Over
Outcome Computation error
Client state Aborted by user
Exit status 203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID 1369231
Run time 24 days 8 hours 7 min 35 sec
CPU time 15 days 16 hours 46 min 55 sec
Validate state Invalid
Credit 8,087.04
Device peak FLOPS 2.55 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>7.2.33</core_client_version>
<![CDATA[
<message>
aborted by user
</message>
<stderr_txt>
iting
21:17:10 (6005): No heartbeat from core client for 30 sec - exiting
21:17:11 (6005): No heartbeat from core client for 30 sec - exiting
21:17:12 (6005): No heartbeat from core client for 30 sec - exiting
21:17:13 (6005): No heartbeat from core client for 30 sec - exiting
21:17:14 (6005): No heartbeat from core client for 30 sec - exiting
21:17:15 (6005): No heartbeat from core client for 30 sec - exiting
02:21:08 (6005): No heartbeat from core client for 30 sec - exiting
02:21:09 (6005): No heartbeat from core client for 30 sec - exiting
02:21:10 (6005): No heartbeat from core client for 30 sec - exiting
02:21:11 (6005): No heartbeat from core client for 30 sec - exiting
02:21:12 (6005): No heartbeat from core client for 30 sec - exiting
02:21:13 (6005): No heartbeat from core client for 30 sec - exiting
02:21:14 (6005): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08ae04e0 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
09:42:09 (3376): No heartbeat from core client for 30 sec - exiting
09:42:10 (3376): No heartbeat from core client for 30 sec - exiting
09:42:11 (3376): No heartbeat from core client for 30 sec - exiting
09:42:12 (3376): No heartbeat from core client for 30 sec - exiting
21:33:58 (3376): No heartbeat from core client for 30 sec - exiting
21:33:59 (3376): No heartbeat from core client for 30 sec - exiting
21:34:00 (3376): No heartbeat from core client for 30 sec - exiting
21:34:01 (3376): No heartbeat from core client for 30 sec - exiting
21:34:02 (3376): No heartbeat from core client for 30 sec - exiting
21:34:03 (3376): No heartbeat from core client for 30 sec - exiting
02:29:27 (3376): No heartbeat from core client for 30 sec - exiting
02:29:28 (3376): No heartbeat from core client for 30 sec - exiting
02:29:29 (3376): No heartbeat from core client for 30 sec - exiting
02:29:30 (3376): No heartbeat from core client for 30 sec - exiting
02:29:31 (3376): No heartbeat from core client for 30 sec - exiting
02:29:32 (3376): No heartbeat from core client for 30 sec - exiting
02:29:33 (3376): No heartbeat from core client for 30 sec - exiting
02:29:34 (3376): No heartbeat from core client for 30 sec - exiting
20:53:13 (3376): No heartbeat from core client for 30 sec - exiting
20:53:14 (3376): No heartbeat from core client for 30 sec - exiting
20:53:15 (3376): No heartbeat from core client for 30 sec - exiting
20:53:16 (3376): No heartbeat from core client for 30 sec - exiting
20:53:17 (3376): No heartbeat from core client for 30 sec - exiting
20:53:18 (3376): No heartbeat from core client for 30 sec - exiting
20:53:19 (3376): No heartbeat from core client for 30 sec - exiting
20:53:20 (3376): No heartbeat from core client for 30 sec - exiting
19:37:13 (3376): No heartbeat from core client for 30 sec - exiting
19:37:14 (3376): No heartbeat from core client for 30 sec - exiting
19:37:15 (3376): No heartbeat from core client for 30 sec - exiting
19:37:16 (3376): No heartbeat from core client for 30 sec - exiting
19:37:17 (3376): No heartbeat from core client for 30 sec - exiting
19:37:18 (3376): No heartbeat from core client for 30 sec - exiting
19:37:19 (3376): No heartbeat from core client for 30 sec - exiting
19:37:20 (3376): No heartbeat from core client for 30 sec - exiting
19:37:21 (3376): No heartbeat from core client for 30 sec - exiting
19:37:22 (3376): No heartbeat from core client for 30 sec - exiting
19:37:23 (3376): No heartbeat from core client for 30 sec - exiting
19:37:24 (3376): No heartbeat from core client for 30 sec - exiting
19:37:25 (3376): No heartbeat from core client for 30 sec - exiting
19:37:26 (3376): No heartbeat from core client for 30 sec - exiting
19:37:27 (3376): No heartbeat from core client for 30 sec - exiting
19:37:28 (3376): No heartbeat from core client for 30 sec - exiting
19:37:29 (3376): No heartbeat from core client for 30 sec - exiting
19:37:30 (3376): No heartbeat from core client for 30 sec - exiting
19:37:31 (3376): No heartbeat from core client for 30 sec - exiting
19:37:32 (3376): No heartbeat from core client for 30 sec - exiting
19:37:33 (3376): No heartbeat from core client for 30 sec - exiting
19:37:34 (3376): No heartbeat from core client for 30 sec - exiting
23:56:47 (3376): No heartbeat from core client for 30 sec - exiting
23:56:49 (3376): No heartbeat from core client for 30 sec - exiting
23:56:50 (3376): No heartbeat from core client for 30 sec - exiting
23:56:51 (3376): No heartbeat from core client for 30 sec - exiting
23:56:52 (3376): No heartbeat from core client for 30 sec - exiting
23:56:53 (3376): No heartbeat from core client for 30 sec - exiting
23:56:54 (3376): No heartbeat from core client for 30 sec - exiting
23:56:55 (3376): No heartbeat from core client for 30 sec - exiting
23:56:56 (3376): No heartbeat from core client for 30 sec - exiting
23:56:57 (3376): No heartbeat from core client for 30 sec - exiting
21:18:25 (3376): No heartbeat from core client for 30 sec - exiting
21:18:27 (3376): No heartbeat from core client for 30 sec - exiting
21:18:28 (3376): No heartbeat from core client for 30 sec - exiting
21:18:29 (3376): No heartbeat from core client for 30 sec - exiting
21:18:30 (3376): No heartbeat from core client for 30 sec - exiting
21:18:31 (3376): No heartbeat from core client for 30 sec - exiting
21:18:32 (3376): No heartbeat from core client for 30 sec - exiting
16:56:35 (3376): No heartbeat from core client for 30 sec - exiting
16:56:36 (3376): No heartbeat from core client for 30 sec - exiting
16:56:37 (3376): No heartbeat from core client for 30 sec - exiting
16:56:38 (3376): No heartbeat from core client for 30 sec - exiting
16:56:39 (3376): No heartbeat from core client for 30 sec - exiting
16:56:40 (3376): No heartbeat from core client for 30 sec - exiting
16:56:41 (3376): No heartbeat from core client for 30 sec - exiting
16:56:42 (3376): No heartbeat from core client for 30 sec - exiting
05:19:35 (3376): No heartbeat from core client for 30 sec - exiting
05:19:36 (3376): No heartbeat from core client for 30 sec - exiting
05:19:37 (3376): No heartbeat from core client for 30 sec - exiting
05:19:38 (3376): No heartbeat from core client for 30 sec - exiting
05:19:39 (3376): No heartbeat from core client for 30 sec - exiting
05:19:40 (3376): No heartbeat from core client for 30 sec - exiting
05:19:41 (3376): No heartbeat from core client for 30 sec - exiting
02:21:41 (3376): No heartbeat from core client for 30 sec - exiting
02:21:43 (3376): No heartbeat from core client for 30 sec - exiting
02:21:44 (3376): No heartbeat from core client for 30 sec - exiting
02:21:45 (3376): No heartbeat from core client for 30 sec - exiting
02:21:46 (3376): No heartbeat from core client for 30 sec - exiting
02:21:47 (3376): No heartbeat from core client for 30 sec - exiting
02:21:48 (3376): No heartbeat from core client for 30 sec - exiting
06:13:30 (3376): No heartbeat from core client for 30 sec - exiting
06:13:31 (3376): No heartbeat from core client for 30 sec - exiting
06:13:32 (3376): No heartbeat from core client for 30 sec - exiting
06:13:33 (3376): No heartbeat from core client for 30 sec - exiting
06:13:34 (3376): No heartbeat from core client for 30 sec - exiting
06:13:35 (3376): No heartbeat from core client for 30 sec - exiting
06:13:36 (3376): No heartbeat from core client for 30 sec - exiting
02:49:02 (3376): No heartbeat from core client for 30 sec - exiting
02:49:03 (3376): No heartbeat from core client for 30 sec - exiting
02:49:04 (3376): No heartbeat from core client for 30 sec - exiting
02:49:05 (3376): No heartbeat from core client for 30 sec - exiting
02:49:06 (3376): No heartbeat from core client for 30 sec - exiting
02:49:07 (3376): No heartbeat from core client for 30 sec - exiting
02:49:08 (3376): No heartbeat from core client for 30 sec - exiting
02:49:09 (3376): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0873e4e0 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
15:50:34 (3682): No heartbeat from core client for 30 sec - exiting
15:50:35 (3682): No heartbeat from core client for 30 sec - exiting
15:50:36 (3682): No heartbeat from core client for 30 sec - exiting
15:50:37 (3682): No heartbeat from core client for 30 sec - exiting
15:50:38 (3682): No heartbeat from core client for 30 sec - exiting
15:50:39 (3682): No heartbeat from core client for 30 sec - exiting
15:50:40 (3682): No heartbeat from core client for 30 sec - exiting
15:50:41 (3682): No heartbeat from core client for 30 sec - exiting
15:50:42 (3682): No heartbeat from core client for 30 sec - exiting
15:50:43 (3682): No heartbeat from core client for 30 sec - exiting
15:50:44 (3682): No heartbeat from core client for 30 sec - exiting
15:50:45 (3682): No heartbeat from core client for 30 sec - exiting
17:16:51 (3682): No heartbeat from core client for 30 sec - exiting
17:16:52 (3682): No heartbeat from core client for 30 sec - exiting
17:16:53 (3682): No heartbeat from core client for 30 sec - exiting
17:16:54 (3682): No heartbeat from core client for 30 sec - exiting
17:16:55 (3682): No heartbeat from core client for 30 sec - exiting
17:16:56 (3682): No heartbeat from core client for 30 sec - exiting
17:16:57 (3682): No heartbeat from core client for 30 sec - exiting
17:16:58 (3682): No heartbeat from core client for 30 sec - exiting
17:16:59 (3682): No heartbeat from core client for 30 sec - exiting
17:17:00 (3682): No heartbeat from core client for 30 sec - exiting
17:17:01 (3682): No heartbeat from core client for 30 sec - exiting
17:17:02 (3682): No heartbeat from core client for 30 sec - exiting
17:17:03 (3682): No heartbeat from core client for 30 sec - exiting
17:17:04 (3682): No heartbeat from core client for 30 sec - exiting
17:17:05 (3682): No heartbeat from core client for 30 sec - exiting
17:17:06 (3682): No heartbeat from core client for 30 sec - exiting
17:17:07 (3682): No heartbeat from core client for 30 sec - exiting
17:17:08 (3682): No heartbeat from core client for 30 sec - exiting
17:17:09 (3682): No heartbeat from core client for 30 sec - exiting
17:17:10 (3682): No heartbeat from core client for 30 sec - exiting
18:11:55 (3682): No heartbeat from core client for 30 sec - exiting
18:11:56 (3682): No heartbeat from core client for 30 sec - exiting
18:11:57 (3682): No heartbeat from core client for 30 sec - exiting
18:11:58 (3682): No heartbeat from core client for 30 sec - exiting
18:11:59 (3682): No heartbeat from core client for 30 sec - exiting
18:12:00 (3682): No heartbeat from core client for 30 sec - exiting
14:18:06 (3682): No heartbeat from core client for 30 sec - exiting
14:18:07 (3682): No heartbeat from core client for 30 sec - exiting
14:18:08 (3682): No heartbeat from core client for 30 sec - exiting
14:18:09 (3682): No heartbeat from core client for 30 sec - exiting
14:18:10 (3682): No heartbeat from core client for 30 sec - exiting
14:18:11 (3682): No heartbeat from core client for 30 sec - exiting
14:18:12 (3682): No heartbeat from core client for 30 sec - exiting
14:18:13 (3682): No heartbeat from core client for 30 sec - exiting
14:18:14 (3682): No heartbeat from core client for 30 sec - exiting
14:18:15 (3682): No heartbeat from core client for 30 sec - exiting
14:18:16 (3682): No heartbeat from core client for 30 sec - exiting
14:18:17 (3682): No heartbeat from core client for 30 sec - exiting
14:18:18 (3682): No heartbeat from core client for 30 sec - exiting
14:18:19 (3682): No heartbeat from core client for 30 sec - exiting
20:18:13 (3682): No heartbeat from core client for 30 sec - exiting
20:18:14 (3682): No heartbeat from core client for 30 sec - exiting
20:18:15 (3682): No heartbeat from core client for 30 sec - exiting
20:18:16 (3682): No heartbeat from core client for 30 sec - exiting
20:18:17 (3682): No heartbeat from core client for 30 sec - exiting
20:18:18 (3682): No heartbeat from core client for 30 sec - exiting
15:12:50 (3682): No heartbeat from core client for 30 sec - exiting
15:12:51 (3682): No heartbeat from core client for 30 sec - exiting
15:12:52 (3682): No heartbeat from core client for 30 sec - exiting
15:12:53 (3682): No heartbeat from core client for 30 sec - exiting
15:12:54 (3682): No heartbeat from core client for 30 sec - exiting
15:12:55 (3682): No heartbeat from core client for 30 sec - exiting
21:06:38 (3682): No heartbeat from core client for 30 sec - exiting
21:06:39 (3682): No heartbeat from core client for 30 sec - exiting
21:06:40 (3682): No heartbeat from core client for 30 sec - exiting
21:06:41 (3682): No heartbeat from core client for 30 sec - exiting
21:06:42 (3682): No heartbeat from core client for 30 sec - exiting
21:06:43 (3682): No heartbeat from core client for 30 sec - exiting
21:06:44 (3682): No heartbeat from core client for 30 sec - exiting
21:06:45 (3682): No heartbeat from core client for 30 sec - exiting
03:20:10 (3682): No heartbeat from core client for 30 sec - exiting
03:20:11 (3682): No heartbeat from core client for 30 sec - exiting
03:20:12 (3682): No heartbeat from core client for 30 sec - exiting
03:20:13 (3682): No heartbeat from core client for 30 sec - exiting
03:20:14 (3682): No heartbeat from core client for 30 sec - exiting
03:20:15 (3682): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x086b44e0 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
23:09:42 (4052): No heartbeat from core client for 30 sec - exiting
23:09:44 (4052): No heartbeat from core client for 30 sec - exiting
23:09:45 (4052): No heartbeat from core client for 30 sec - exiting
23:09:46 (4052): No heartbeat from core client for 30 sec - exiting
23:09:47 (4052): No heartbeat from core client for 30 sec - exiting
23:09:48 (4052): No heartbeat from core client for 30 sec - exiting
23:09:49 (4052): No heartbeat from core client for 30 sec - exiting
23:09:50 (4052): No heartbeat from core client for 30 sec - exiting
05:24:19 (4052): No heartbeat from core client for 30 sec - exiting
05:24:20 (4052): No heartbeat from core client for 30 sec - exiting
05:24:21 (4052): No heartbeat from core client for 30 sec - exiting
05:24:22 (4052): No heartbeat from core client for 30 sec - exiting
05:24:23 (4052): No heartbeat from core client for 30 sec - exiting
05:24:24 (4052): No heartbeat from core client for 30 sec - exiting
00:22:43 (4052): No heartbeat from core client for 30 sec - exiting
00:22:44 (4052): No heartbeat from core client for 30 sec - exiting
00:22:45 (4052): No heartbeat from core client for 30 sec - exiting
00:22:46 (4052): No heartbeat from core client for 30 sec - exiting
00:22:47 (4052): No heartbeat from core client for 30 sec - exiting
00:22:48 (4052): No heartbeat from core client for 30 sec - exiting
00:22:49 (4052): No heartbeat from core client for 30 sec - exiting
21:39:40 (4052): No heartbeat from core client for 30 sec - exiting
21:39:41 (4052): No heartbeat from core client for 30 sec - exiting
21:39:42 (4052): No heartbeat from core client for 30 sec - exiting
21:39:43 (4052): No heartbeat from core client for 30 sec - exiting
21:39:44 (4052): No heartbeat from core client for 30 sec - exiting
02:08:22 (4052): No heartbeat from core client for 30 sec - exiting
02:08:23 (4052): No heartbeat from core client for 30 sec - exiting
02:08:24 (4052): No heartbeat from core client for 30 sec - exiting
02:08:25 (4052): No heartbeat from core client for 30 sec - exiting
02:08:26 (4052): No heartbeat from core client for 30 sec - exiting
02:08:27 (4052): No heartbeat from core client for 30 sec - exiting
02:08:28 (4052): No heartbeat from core client for 30 sec - exiting
02:08:29 (4052): No heartbeat from core client for 30 sec - exiting
11:02:19 (4052): No heartbeat from core client for 30 sec - exiting
11:02:20 (4052): No heartbeat from core client for 30 sec - exiting
11:02:21 (4052): No heartbeat from core client for 30 sec - exiting
11:02:22 (4052): No heartbeat from core client for 30 sec - exiting
11:02:23 (4052): No heartbeat from core client for 30 sec - exiting
11:02:24 (4052): No heartbeat from core client for 30 sec - exiting
11:02:25 (4052): No heartbeat from core client for 30 sec - exiting
01:10:00 (4052): No heartbeat from core client for 30 sec - exiting
01:10:01 (4052): No heartbeat from core client for 30 sec - exiting
01:10:02 (4052): No heartbeat from core client for 30 sec - exiting
01:10:03 (4052): No heartbeat from core client for 30 sec - exiting
01:10:04 (4052): No heartbeat from core client for 30 sec - exiting
01:10:05 (4052): No heartbeat from core client for 30 sec - exiting
01:10:06 (4052): No heartbeat from core client for 30 sec - exiting
01:10:07 (4052): No heartbeat from core client for 30 sec - exiting
20:25:58 (4052): No heartbeat from core client for 30 sec - exiting
20:25:59 (4052): No heartbeat from core client for 30 sec - exiting
20:26:00 (4052): No heartbeat from core client for 30 sec - exiting
20:26:01 (4052): No heartbeat from core client for 30 sec - exiting
20:26:02 (4052): No heartbeat from core client for 30 sec - exiting
20:26:03 (4052): No heartbeat from core client for 30 sec - exiting
20:26:04 (4052): No heartbeat from core client for 30 sec - exiting
20:26:05 (4052): No heartbeat from core client for 30 sec - exiting
20:26:06 (4052): No heartbeat from core client for 30 sec - exiting
20:26:07 (4052): No heartbeat from core client for 30 sec - exiting
20:26:08 (4052): No heartbeat from core client for 30 sec - exiting
20:26:09 (4052): No heartbeat from core client for 30 sec - exiting
20:26:10 (4052): No heartbeat from core client for 30 sec - exiting
20:26:11 (4052): No heartbeat from core client for 30 sec - exiting
20:26:12 (4052): No heartbeat from core client for 30 sec - exiting
20:26:13 (4052): No heartbeat from core client for 30 sec - exiting
20:26:14 (4052): No heartbeat from core client for 30 sec - exiting
06:09:13 (4052): No heartbeat from core client for 30 sec - exiting
06:09:15 (4052): No heartbeat from core client for 30 sec - exiting
06:09:16 (4052): No heartbeat from core client for 30 sec - exiting
06:09:17 (4052): No heartbeat from core client for 30 sec - exiting
06:09:18 (4052): No heartbeat from core client for 30 sec - exiting
06:09:19 (4052): No heartbeat from core client for 30 sec - exiting
20:29:28 (4052): No heartbeat from core client for 30 sec - exiting
20:29:29 (4052): No heartbeat from core client for 30 sec - exiting
20:29:30 (4052): No heartbeat from core client for 30 sec - exiting
20:29:31 (4052): No heartbeat from core client for 30 sec - exiting
20:29:33 (4052): No heartbeat from core client for 30 sec - exiting
20:29:34 (4052): No heartbeat from core client for 30 sec - exiting
20:29:35 (4052): No heartbeat from core client for 30 sec - exiting
20:29:36 (4052): No heartbeat from core client for 30 sec - exiting
20:29:37 (4052): No heartbeat from core client for 30 sec - exiting
20:29:38 (4052): No heartbeat from core client for 30 sec - exiting
20:29:39 (4052): No heartbeat from core client for 30 sec - exiting
20:29:40 (4052): No heartbeat from core client for 30 sec - exiting
20:29:41 (4052): No heartbeat from core client for 30 sec - exiting
20:29:42 (4052): No heartbeat from core client for 30 sec - exiting
20:29:43 (4052): No heartbeat from core client for 30 sec - exiting
20:29:44 (4052): No heartbeat from core client for 30 sec - exiting
20:29:45 (4052): No heartbeat from core client for 30 sec - exiting
20:29:46 (4052): No heartbeat from core client for 30 sec - exiting
20:29:47 (4052): No heartbeat from core client for 30 sec - exiting
20:29:48 (4052): No heartbeat from core client for 30 sec - exiting
20:29:49 (4052): No heartbeat from core client for 30 sec - exiting
20:29:50 (4052): No heartbeat from core client for 30 sec - exiting
20:29:51 (4052): No heartbeat from core client for 30 sec - exiting
20:29:52 (4052): No heartbeat from core client for 30 sec - exiting
20:29:53 (4052): No heartbeat from core client for 30 sec - exiting
20:29:54 (4052): No heartbeat from core client for 30 sec - exiting
20:29:55 (4052): No heartbeat from core client for 30 sec - exiting
20:29:56 (4052): No heartbeat from core client for 30 sec - exiting
20:29:57 (4052): No heartbeat from core client for 30 sec - exiting
20:29:58 (4052): No heartbeat from core client for 30 sec - exiting
20:29:59 (4052): No heartbeat from core client for 30 sec - exiting
20:30:00 (4052): No heartbeat from core client for 30 sec - exiting
20:30:01 (4052): No heartbeat from core client for 30 sec - exiting
20:30:02 (4052): No heartbeat from core client for 30 sec - exiting
20:30:03 (4052): No heartbeat from core client for 30 sec - exiting
20:30:04 (4052): No heartbeat from core client for 30 sec - exiting
20:30:05 (4052): No heartbeat from core client for 30 sec - exiting
20:30:06 (4052): No heartbeat from core client for 30 sec - exiting
20:30:07 (4052): No heartbeat from core client for 30 sec - exiting
20:30:09 (4052): No heartbeat from core client for 30 sec - exiting
20:30:10 (4052): No heartbeat from core client for 30 sec - exiting
20:30:11 (4052): No heartbeat from core client for 30 sec - exiting
20:30:12 (4052): No heartbeat from core client for 30 sec - exiting
20:30:13 (4052): No heartbeat from core client for 30 sec - exiting
20:30:14 (4052): No heartbeat from core client for 30 sec - exiting
20:30:15 (4052): No heartbeat from core client for 30 sec - exiting
20:30:16 (4052): No heartbeat from core client for 30 sec - exiting
20:30:17 (4052): No heartbeat from core client for 30 sec - exiting
20:30:18 (4052): No heartbeat from core client for 30 sec - exiting
20:30:19 (4052): No heartbeat from core client for 30 sec - exiting
20:30:20 (4052): No heartbeat from core client for 30 sec - exiting
20:30:21 (4052): No heartbeat from core client for 30 sec - exiting
20:30:22 (4052): No heartbeat from core client for 30 sec - exiting
20:30:23 (4052): No heartbeat from core client for 30 sec - exiting
20:30:24 (4052): No heartbeat from core client for 30 sec - exiting
20:30:25 (4052): No heartbeat from core client for 30 sec - exiting
20:30:26 (4052): No heartbeat from core client for 30 sec - exiting
20:30:27 (4052): No heartbeat from core client for 30 sec - exiting
20:30:28 (4052): No heartbeat from core client for 30 sec - exiting
20:30:29 (4052): No heartbeat from core client for 30 sec - exiting
20:30:31 (4052): No heartbeat from core client for 30 sec - exiting
20:30:32 (4052): No heartbeat from core client for 30 sec - exiting
20:30:33 (4052): No heartbeat from core client for 30 sec - exiting
20:30:34 (4052): No heartbeat from core client for 30 sec - exiting
20:30:36 (4052): No heartbeat from core client for 30 sec - exiting
20:30:37 (4052): No heartbeat from core client for 30 sec - exiting
20:30:38 (4052): No heartbeat from core client for 30 sec - exiting
20:30:39 (4052): No heartbeat from core client for 30 sec - exiting
20:30:41 (4052): No heartbeat from core client for 30 sec - exiting
20:30:42 (4052): No heartbeat from core client for 30 sec - exiting
20:30:43 (4052): No heartbeat from core client for 30 sec - exiting
20:30:44 (4052): No heartbeat from core client for 30 sec - exiting
20:30:45 (4052): No heartbeat from core client for 30 sec - exiting
20:30:46 (4052): No heartbeat from core client for 30 sec - exiting
20:30:47 (4052): No heartbeat from core client for 30 sec - exiting
20:30:48 (4052): No heartbeat from core client for 30 sec - exiting
20:30:49 (4052): No heartbeat from core client for 30 sec - exiting
20:30:50 (4052): No heartbeat from core client for 30 sec - exiting
20:30:51 (4052): No heartbeat from core client for 30 sec - exiting
20:30:52 (4052): No heartbeat from core client for 30 sec - exiting
20:30:53 (4052): No heartbeat from core client for 30 sec - exiting
20:30:54 (4052): No heartbeat from core client for 30 sec - exiting
20:30:55 (4052): No heartbeat from core client for 30 sec - exiting
20:30:56 (4052): No heartbeat from core client for 30 sec - exiting
20:30:57 (4052): No heartbeat from core client for 30 sec - exiting
20:30:58 (4052): No heartbeat from core client for 30 sec - exiting
20:30:59 (4052): No heartbeat from core client for 30 sec - exiting
20:31:00 (4052): No heartbeat from core client for 30 sec - exiting
20:31:01 (4052): No heartbeat from core client for 30 sec - exiting
20:31:02 (4052): No heartbeat from core client for 30 sec - exiting
20:31:03 (4052): No heartbeat from core client for 30 sec - exiting
20:31:04 (4052): No heartbeat from core client for 30 sec - exiting
20:31:05 (4052): No heartbeat from core client for 30 sec - exiting
21:35:39 (4052): No heartbeat from core client for 30 sec - exiting
21:35:41 (4052): No heartbeat from core client for 30 sec - exiting
21:35:42 (4052): No heartbeat from core client for 30 sec - exiting
21:35:43 (4052): No heartbeat from core client for 30 sec - exiting
21:35:44 (4052): No heartbeat from core client for 30 sec - exiting
21:35:57 (4052): No heartbeat from core client for 30 sec - exiting
21:35:58 (4052): No heartbeat from core client for 30 sec - exiting
21:35:59 (4052): No heartbeat from core client for 30 sec - exiting
21:36:00 (4052): No heartbeat from core client for 30 sec - exiting
21:36:01 (4052): No heartbeat from core client for 30 sec - exiting
21:36:02 (4052): No heartbeat from core client for 30 sec - exiting
21:36:03 (4052): No heartbeat from core client for 30 sec - exiting
21:36:04 (4052): No heartbeat from core client for 30 sec - exiting
21:36:05 (4052): No heartbeat from core client for 30 sec - exiting
21:36:06 (4052): No heartbeat from core client for 30 sec - exiting
21:36:07 (4052): No heartbeat from core client for 30 sec - exiting
21:36:08 (4052): No heartbeat from core client for 30 sec - exiting
21:36:09 (4052): No heartbeat from core client for 30 sec - exiting
21:36:10 (4052): No heartbeat from core client for 30 sec - exiting
21:36:11 (4052): No heartbeat from core client for 30 sec - exiting
21:36:12 (4052): No heartbeat from core client for 30 sec - exiting
21:36:13 (4052): No heartbeat from core client for 30 sec - exiting
21:36:14 (4052): No heartbeat from core client for 30 sec - exiting
21:36:15 (4052): No heartbeat from core client for 30 sec - exiting
21:36:16 (4052): No heartbeat from core client for 30 sec - exiting
21:36:17 (4052): No heartbeat from core client for 30 sec - exiting
21:36:18 (4052): No heartbeat from core client for 30 sec - exiting
21:36:19 (4052): No heartbeat from core client for 30 sec - exiting
21:36:20 (4052): No heartbeat from core client for 30 sec - exiting
21:36:21 (4052): No heartbeat from core client for 30 sec - exiting
21:36:22 (4052): No heartbeat from core client for 30 sec - exiting
21:36:23 (4052): No heartbeat from core client for 30 sec - exiting
21:36:24 (4052): No heartbeat from core client for 30 sec - exiting
21:36:25 (4052): No heartbeat from core client for 30 sec - exiting
21:36:26 (4052): No heartbeat from core client for 30 sec - exiting
21:36:27 (4052): No heartbeat from core client for 30 sec - exiting
21:36:28 (4052): No heartbeat from core client for 30 sec - exiting
21:36:29 (4052): No heartbeat from core client for 30 sec - exiting
21:36:30 (4052): No heartbeat from core client for 30 sec - exiting
21:36:31 (4052): No heartbeat from core client for 30 sec - exiting
21:36:32 (4052): No heartbeat from core client for 30 sec - exiting
21:36:33 (4052): No heartbeat from core client for 30 sec - exiting
21:36:34 (4052): No heartbeat from core client for 30 sec - exiting
21:36:35 (4052): No heartbeat from core client for 30 sec - exiting
21:36:36 (4052): No heartbeat from core client for 30 sec - exiting
21:36:37 (4052): No heartbeat from core client for 30 sec - exiting
21:36:38 (4052): No heartbeat from core client for 30 sec - exiting
21:36:39 (4052): No heartbeat from core client for 30 sec - exiting
21:36:40 (4052): No heartbeat from core client for 30 sec - exiting
21:36:42 (4052): No heartbeat from core client for 30 sec - exiting
21:36:43 (4052): No heartbeat from core client for 30 sec - exiting
21:36:44 (4052): No heartbeat from core client for 30 sec - exiting
21:36:45 (4052): No heartbeat from core client for 30 sec - exiting
21:36:46 (4052): No heartbeat from core client for 30 sec - exiting
21:36:47 (4052): No heartbeat from core client for 30 sec - exiting
21:36:48 (4052): No heartbeat from core client for 30 sec - exiting
21:36:49 (4052): No heartbeat from core client for 30 sec - exiting
21:36:50 (4052): No heartbeat from core client for 30 sec - exiting
21:36:51 (4052): No heartbeat from core client for 30 sec - exiting
21:36:52 (4052): No heartbeat from core client for 30 sec - exiting
21:36:53 (4052): No heartbeat from core client for 30 sec - exiting
21:36:54 (4052): No heartbeat from core client for 30 sec - exiting
21:36:55 (4052): No heartbeat from core client for 30 sec - exiting
21:36:56 (4052): No heartbeat from core client for 30 sec - exiting
21:36:57 (4052): No heartbeat from core client for 30 sec - exiting
21:36:58 (4052): No heartbeat from core client for 30 sec - exiting
21:36:59 (4052): No heartbeat from core client for 30 sec - exiting
21:37:00 (4052): No heartbeat from core client for 30 sec - exiting
21:37:01 (4052): No heartbeat from core client for 30 sec - exiting
21:37:02 (4052): No heartbeat from core client for 30 sec - exiting
21:37:03 (4052): No heartbeat from core client for 30 sec - exiting
21:37:04 (4052): No heartbeat from core client for 30 sec - exiting
21:37:05 (4052): No heartbeat from core client for 30 sec - exiting
21:37:06 (4052): No heartbeat from core client for 30 sec - exiting
21:37:08 (4052): No heartbeat from core client for 30 sec - exiting
21:37:09 (4052): No heartbeat from core client for 30 sec - exiting
21:37:10 (4052): No heartbeat from core client for 30 sec - exiting
21:37:11 (4052): No heartbeat from core client for 30 sec - exiting
21:37:13 (4052): No heartbeat from core client for 30 sec - exiting
21:37:14 (4052): No heartbeat from core client for 30 sec - exiting
21:37:15 (4052): No heartbeat from core client for 30 sec - exiting
21:37:16 (4052): No heartbeat from core client for 30 sec - exiting
21:37:17 (4052): No heartbeat from core client for 30 sec - exiting
21:37:18 (4052): No heartbeat from core client for 30 sec - exiting
21:37:19 (4052): No heartbeat from core client for 30 sec - exiting
21:37:20 (4052): No heartbeat from core client for 30 sec - exiting
21:37:21 (4052): No heartbeat from core client for 30 sec - exiting
21:37:23 (4052): No heartbeat from core client for 30 sec - exiting
21:37:24 (4052): No heartbeat from core client for 30 sec - exiting
21:37:25 (4052): No heartbeat from core client for 30 sec - exiting
21:37:26 (4052): No heartbeat from core client for 30 sec - exiting
21:37:27 (4052): No heartbeat from core client for 30 sec - exiting
21:37:28 (4052): No heartbeat from core client for 30 sec - exiting
21:37:29 (4052): No heartbeat from core client for 30 sec - exiting
21:37:30 (4052): No heartbeat from core client for 30 sec - exiting
21:37:31 (4052): No heartbeat from core client for 30 sec - exiting
21:37:33 (4052): No heartbeat from core client for 30 sec - exiting
21:37:34 (4052): No heartbeat from core client for 30 sec - exiting
21:37:35 (4052): No heartbeat from core client for 30 sec - exiting
21:37:36 (4052): No heartbeat from core client for 30 sec - exiting
21:37:38 (4052): No heartbeat from core client for 30 sec - exiting
21:37:39 (4052): No heartbeat from core client for 30 sec - exiting
21:37:40 (4052): No heartbeat from core client for 30 sec - exiting
21:37:42 (4052): No heartbeat from core client for 30 sec - exiting
21:37:43 (4052): No heartbeat from core client for 30 sec - exiting
21:37:44 (4052): No heartbeat from core client for 30 sec - exiting
21:37:45 (4052): No heartbeat from core client for 30 sec - exiting
21:37:46 (4052): No heartbeat from core client for 30 sec - exiting
21:37:47 (4052): No heartbeat from core client for 30 sec - exiting
21:37:48 (4052): No heartbeat from core client for 30 sec - exiting
21:37:49 (4052): No heartbeat from core client for 30 sec - exiting
21:37:50 (4052): No heartbeat from core client for 30 sec - exiting
21:37:51 (4052): No heartbeat from core client for 30 sec - exiting
21:37:52 (4052): No heartbeat from core client for 30 sec - exiting
21:37:53 (4052): No heartbeat from core client for 30 sec - exiting
21:37:54 (4052): No heartbeat from core client for 30 sec - exiting
21:37:55 (4052): No heartbeat from core client for 30 sec - exiting
21:37:56 (4052): No heartbeat from core client for 30 sec - exiting
21:37:57 (4052): No heartbeat from core client for 30 sec - exiting
21:37:58 (4052): No heartbeat from core client for 30 sec - exiting
21:37:59 (4052): No heartbeat from core client for 30 sec - exiting
21:38:00 (4052): No heartbeat from core client for 30 sec - exiting
21:38:01 (4052): No heartbeat from core client for 30 sec - exiting
21:38:02 (4052): No heartbeat from core client for 30 sec - exiting
21:38:03 (4052): No heartbeat from core client for 30 sec - exiting
21:38:04 (4052): No heartbeat from core client for 30 sec - exiting
21:38:05 (4052): No heartbeat from core client for 30 sec - exiting
21:38:06 (4052): No heartbeat from core client for 30 sec - exiting
21:38:07 (4052): No heartbeat from core client for 30 sec - exiting
21:38:09 (4052): No heartbeat from core client for 30 sec - exiting
21:38:10 (4052): No heartbeat from core client for 30 sec - exiting
21:38:11 (4052): No heartbeat from core client for 30 sec - exiting
21:38:12 (4052): No heartbeat from core client for 30 sec - exiting
21:38:13 (4052): No heartbeat from core client for 30 sec - exiting
21:38:15 (4052): No heartbeat from core client for 30 sec - exiting
21:38:16 (4052): No heartbeat from core client for 30 sec - exiting
21:38:17 (4052): No heartbeat from core client for 30 sec - exiting
21:38:19 (4052): No heartbeat from core client for 30 sec - exiting
21:38:20 (4052): No heartbeat from core client for 30 sec - exiting
21:38:21 (4052): No heartbeat from core client for 30 sec - exiting
21:38:22 (4052): No heartbeat from core client for 30 sec - exiting
21:38:24 (4052): No heartbeat from core client for 30 sec - exiting
21:38:25 (4052): No heartbeat from core client for 30 sec - exiting
21:38:26 (4052): No heartbeat from core client for 30 sec - exiting
21:38:27 (4052): No heartbeat from core client for 30 sec - exiting
21:38:28 (4052): No heartbeat from core client for 30 sec - exiting
21:38:29 (4052): No heartbeat from core client for 30 sec - exiting
21:38:30 (4052): No heartbeat from core client for 30 sec - exiting
21:38:31 (4052): No heartbeat from core client for 30 sec - exiting
21:38:32 (4052): No heartbeat from core client for 30 sec - exiting
21:38:33 (4052): No heartbeat from core client for 30 sec - exiting
21:38:34 (4052): No heartbeat from core client for 30 sec - exiting
21:38:35 (4052): No heartbeat from core client for 30 sec - exiting
21:38:36 (4052): No heartbeat from core client for 30 sec - exiting
21:38:37 (4052): No heartbeat from core client for 30 sec - exiting
21:38:38 (4052): No heartbeat from core client for 30 sec - exiting
21:38:39 (4052): No heartbeat from core client for 30 sec - exiting
21:38:40 (4052): No heartbeat from core client for 30 sec - exiting
21:38:41 (4052): No heartbeat from core client for 30 sec - exiting
21:38:42 (4052): No heartbeat from core client for 30 sec - exiting
21:38:43 (4052): No heartbeat from core client for 30 sec - exiting
21:38:44 (4052): No heartbeat from core client for 30 sec - exiting
21:38:45 (4052): No heartbeat from core client for 30 sec - exiting
21:38:46 (4052): No heartbeat from core client for 30 sec - exiting
21:38:47 (4052): No heartbeat from core client for 30 sec - exiting
21:38:48 (4052): No heartbeat from core client for 30 sec - exiting
21:38:49 (4052): No heartbeat from core client for 30 sec - exiting
21:38:51 (4052): No heartbeat from core client for 30 sec - exiting
21:38:52 (4052): No heartbeat from core client for 30 sec - exiting
21:38:53 (4052): No heartbeat from core client for 30 sec - exiting
21:38:54 (4052): No heartbeat from core client for 30 sec - exiting
21:38:55 (4052): No heartbeat from core client for 30 sec - exiting
21:38:56 (4052): No heartbeat from core client for 30 sec - exiting
21:38:57 (4052): No heartbeat from core client for 30 sec - exiting
21:38:58 (4052): No heartbeat from core client for 30 sec - exiting
21:38:59 (4052): No heartbeat from core client for 30 sec - exiting
21:39:01 (4052): No heartbeat from core client for 30 sec - exiting
21:39:02 (4052): No heartbeat from core client for 30 sec - exiting
21:39:03 (4052): No heartbeat from core client for 30 sec - exiting
21:39:05 (4052): No heartbeat from core client for 30 sec - exiting
21:39:06 (4052): No heartbeat from core client for 30 sec - exiting
21:39:07 (4052): No heartbeat from core client for 30 sec - exiting
21:39:08 (4052): No heartbeat from core client for 30 sec - exiting
21:39:09 (4052): No heartbeat from core client for 30 sec - exiting
21:39:10 (4052): No heartbeat from core client for 30 sec - exiting
21:39:11 (4052): No heartbeat from core client for 30 sec - exiting
21:39:12 (4052): No heartbeat from core client for 30 sec - exiting
21:39:13 (4052): No heartbeat from core client for 30 sec - exiting
21:39:14 (4052): No heartbeat from core client for 30 sec - exiting
21:39:16 (4052): No heartbeat from core client for 30 sec - exiting
21:39:17 (4052): No heartbeat from core client for 30 sec - exiting
21:39:18 (4052): No heartbeat from core client for 30 sec - exiting
21:39:19 (4052): No heartbeat from core client for 30 sec - exiting
21:39:22 (4052): No heartbeat from core client for 30 sec - exiting
21:39:23 (4052): No heartbeat from core client for 30 sec - exiting
21:39:24 (4052): No heartbeat from core client for 30 sec - exiting
21:39:25 (4052): No heartbeat from core client for 30 sec - exiting
21:39:26 (4052): No heartbeat from core client for 30 sec - exiting
21:39:27 (4052): No heartbeat from core client for 30 sec - exiting
21:39:28 (4052): No heartbeat from core client for 30 sec - exiting
21:39:29 (4052): No heartbeat from core client for 30 sec - exiting
21:39:30 (4052): No heartbeat from core client for 30 sec - exiting
21:39:31 (4052): No heartbeat from core client for 30 sec - exiting
21:39:32 (4052): No heartbeat from core client for 30 sec - exiting
21:39:33 (4052): No heartbeat from core client for 30 sec - exiting
21:39:34 (4052): No heartbeat from core client for 30 sec - exiting
06:58:30 (4052): No heartbeat from core client for 30 sec - exiting
06:58:31 (4052): No heartbeat from core client for 30 sec - exiting
06:58:32 (4052): No heartbeat from core client for 30 sec - exiting
06:58:33 (4052): No heartbeat from core client for 30 sec - exiting
06:58:34 (4052): No heartbeat from core client for 30 sec - exiting
06:58:35 (4052): No heartbeat from core client for 30 sec - exiting
06:58:36 (4052): No heartbeat from core client for 30 sec - exiting
06:58:37 (4052): No heartbeat from core client for 30 sec - exiting
06:58:38 (4052): No heartbeat from core client for 30 sec - exiting
06:58:39 (4052): No heartbeat from core client for 30 sec - exiting
01:18:56 (4052): No heartbeat from core client for 30 sec - exiting
01:18:57 (4052): No heartbeat from core client for 30 sec - exiting
01:18:58 (4052): No heartbeat from core client for 30 sec - exiting
01:18:59 (4052): No heartbeat from core client for 30 sec - exiting
01:19:00 (4052): No heartbeat from core client for 30 sec - exiting
01:19:01 (4052): No heartbeat from core client for 30 sec - exiting
01:19:02 (4052): No heartbeat from core client for 30 sec - exiting
01:19:03 (4052): No heartbeat from core client for 30 sec - exiting
10:53:27 (4052): No heartbeat from core client for 30 sec - exiting
10:53:28 (4052): No heartbeat from core client for 30 sec - exiting
10:53:29 (4052): No heartbeat from core client for 30 sec - exiting
10:53:30 (4052): No heartbeat from core client for 30 sec - exiting
10:53:31 (4052): No heartbeat from core client for 30 sec - exiting
10:53:32 (4052): No heartbeat from core client for 30 sec - exiting
17:43:30 (4052): No heartbeat from core client for 30 sec - exiting
17:43:31 (4052): No heartbeat from core client for 30 sec - exiting
17:43:32 (4052): No heartbeat from core client for 30 sec - exiting
17:43:33 (4052): No heartbeat from core client for 30 sec - exiting
17:43:34 (4052): No heartbeat from core client for 30 sec - exiting
17:43:35 (4052): No heartbeat from core client for 30 sec - exiting
17:43:36 (4052): No heartbeat from core client for 30 sec - exiting
17:43:37 (4052): No heartbeat from core client for 30 sec - exiting
11:11:41 (4052): No heartbeat from core client for 30 sec - exiting
11:11:43 (4052): No heartbeat from core client for 30 sec - exiting
11:11:44 (4052): No heartbeat from core client for 30 sec - exiting
11:11:45 (4052): No heartbeat from core client for 30 sec - exiting
11:11:46 (4052): No heartbeat from core client for 30 sec - exiting
11:11:47 (4052): No heartbeat from core client for 30 sec - exiting
01:02:24 (4052): No heartbeat from core client for 30 sec - exiting
01:02:26 (4052): No heartbeat from core client for 30 sec - exiting
01:02:27 (4052): No heartbeat from core client for 30 sec - exiting
01:02:28 (4052): No heartbeat from core client for 30 sec - exiting
01:02:29 (4052): No heartbeat from core client for 30 sec - exiting
01:02:30 (4052): No heartbeat from core client for 30 sec - exiting
01:02:31 (4052): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0882f4e0 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
18:00:02 (8983): No heartbeat from core client for 30 sec - exiting
18:00:03 (8983): No heartbeat from core client for 30 sec - exiting
18:00:04 (8983): No heartbeat from core client for 30 sec - exiting
18:00:05 (8983): No heartbeat from core client for 30 sec - exiting
18:00:06 (8983): No heartbeat from core client for 30 sec - exiting
22:34:03 (8983): No heartbeat from core client for 30 sec - exiting
22:34:04 (8983): No heartbeat from core client for 30 sec - exiting
22:34:05 (8983): No heartbeat from core client for 30 sec - exiting
22:34:06 (8983): No heartbeat from core client for 30 sec - exiting
22:34:07 (8983): No heartbeat from core client for 30 sec - exiting
22:34:08 (8983): No heartbeat from core client for 30 sec - exiting
22:34:09 (8983): No heartbeat from core client for 30 sec - exiting
22:34:10 (8983): No heartbeat from core client for 30 sec - exiting
14:48:35 (8983): No heartbeat from core client for 30 sec - exiting
14:48:36 (8983): No heartbeat from core client for 30 sec - exiting
14:48:37 (8983): No heartbeat from core client for 30 sec - exiting
14:48:38 (8983): No heartbeat from core client for 30 sec - exiting
14:48:39 (8983): No heartbeat from core client for 30 sec - exiting
14:48:40 (8983): No heartbeat from core client for 30 sec - exiting
14:48:41 (8983): No heartbeat from core client for 30 sec - exiting
14:48:42 (8983): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x099674e0 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
00:59:57 (5574): No heartbeat from core client for 30 sec - exiting
00:59:58 (5574): No heartbeat from core client for 30 sec - exiting
00:59:59 (5574): No heartbeat from core client for 30 sec - exiting
01:00:00 (5574): No heartbeat from core client for 30 sec - exiting
01:00:01 (5574): No heartbeat from core client for 30 sec - exiting
01:00:02 (5574): No heartbeat from core client for 30 sec - exiting
01:00:03 (5574): No heartbeat from core client for 30 sec - exiting
01:00:04 (5574): No heartbeat from core client for 30 sec - exiting
19:39:31 (5574): No heartbeat from core client for 30 sec - exiting
19:39:32 (5574): No heartbeat from core client for 30 sec - exiting
19:39:33 (5574): No heartbeat from core client for 30 sec - exiting
19:39:34 (5574): No heartbeat from core client for 30 sec - exiting
19:39:35 (5574): No heartbeat from core client for 30 sec - exiting
19:39:36 (5574): No heartbeat from core client for 30 sec - exiting
19:39:37 (5574): No heartbeat from core client for 30 sec - exiting
00:35:22 (5574): No heartbeat from core client for 30 sec - exiting
00:35:24 (5574): No heartbeat from core client for 30 sec - exiting
00:35:25 (5574): No heartbeat from core client for 30 sec - exiting
00:35:26 (5574): No heartbeat from core client for 30 sec - exiting
00:35:27 (5574): No heartbeat from core client for 30 sec - exiting
00:35:28 (5574): No heartbeat from core client for 30 sec - exiting
00:35:29 (5574): No heartbeat from core client for 30 sec - exiting
00:35:30 (5574): No heartbeat from core client for 30 sec - exiting
09:44:54 (5574): No heartbeat from core client for 30 sec - exiting
09:44:55 (5574): No heartbeat from core client for 30 sec - exiting
09:44:56 (5574): No heartbeat from core client for 30 sec - exiting
09:44:57 (5574): No heartbeat from core client for 30 sec - exiting
09:44:58 (5574): No heartbeat from core client for 30 sec - exiting
09:44:59 (5574): No heartbeat from core client for 30 sec - exiting
09:45:00 (5574): No heartbeat from core client for 30 sec - exiting
09:45:01 (5574): No heartbeat from core client for 30 sec - exiting
09:45:02 (5574): No heartbeat from core client for 30 sec - exiting
01:15:23 (5574): No heartbeat from core client for 30 sec - exiting
01:15:24 (5574): No heartbeat from core client for 30 sec - exiting
01:15:25 (5574): No heartbeat from core client for 30 sec - exiting
01:15:26 (5574): No heartbeat from core client for 30 sec - exiting
01:15:27 (5574): No heartbeat from core client for 30 sec - exiting
01:15:28 (5574): No heartbeat from core client for 30 sec - exiting
01:15:29 (5574): No heartbeat from core client for 30 sec - exiting
01:15:30 (5574): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09a924e0 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
22:46:09 (3504): No heartbeat from core client for 30 sec - exiting
22:46:10 (3504): No heartbeat from core client for 30 sec - exiting
22:46:11 (3504): No heartbeat from core client for 30 sec - exiting
22:46:12 (3504): No heartbeat from core client for 30 sec - exiting
22:46:13 (3504): No heartbeat from core client for 30 sec - exiting
22:46:14 (3504): No heartbeat from core client for 30 sec - exiting
22:46:15 (3504): No heartbeat from core client for 30 sec - exiting
22:46:16 (3504): No heartbeat from core client for 30 sec - exiting
22:46:17 (3504): No heartbeat from core client for 30 sec - exiting
22:46:18 (3504): No heartbeat from core client for 30 sec - exiting
22:46:19 (3504): No heartbeat from core client for 30 sec - exiting
22:46:20 (3504): No heartbeat from core client for 30 sec - exiting
22:46:21 (3504): No heartbeat from core client for 30 sec - exiting
22:46:22 (3504): No heartbeat from core client for 30 sec - exiting
22:46:23 (3504): No heartbeat from core client for 30 sec - exiting
22:46:24 (3504): No heartbeat from core client for 30 sec - exiting
22:46:25 (3504): No heartbeat from core client for 30 sec - exiting
22:46:26 (3504): No heartbeat from core client for 30 sec - exiting
22:46:27 (3504): No heartbeat from core client for 30 sec - exiting
22:46:28 (3504): No heartbeat from core client for 30 sec - exiting
22:46:29 (3504): No heartbeat from core client for 30 sec - exiting
22:46:30 (3504): No heartbeat from core client for 30 sec - exiting
22:46:31 (3504): No heartbeat from core client for 30 sec - exiting
23:03:43 (3504): No heartbeat from core client for 30 sec - exiting
23:03:44 (3504): No heartbeat from core client for 30 sec - exiting
23:03:45 (3504): No heartbeat from core client for 30 sec - exiting
23:03:46 (3504): No heartbeat from core client for 30 sec - exiting
23:03:47 (3504): No heartbeat from core client for 30 sec - exiting
23:03:48 (3504): No heartbeat from core client for 30 sec - exiting
23:03:49 (3504): No heartbeat from core client for 30 sec - exiting
23:03:50 (3504): No heartbeat from core client for 30 sec - exiting
03:30:18 (3504): No heartbeat from core client for 30 sec - exiting
03:30:19 (3504): No heartbeat from core client for 30 sec - exiting
19:34:29 (3504): No heartbeat from core client for 30 sec - exiting
19:34:30 (3504): No heartbeat from core client for 30 sec - exiting
19:34:31 (3504): No heartbeat from core client for 30 sec - exiting
19:34:32 (3504): No heartbeat from core client for 30 sec - exiting
19:34:33 (3504): No heartbeat from core client for 30 sec - exiting
19:34:34 (3504): No heartbeat from core client for 30 sec - exiting
19:34:35 (3504): No heartbeat from core client for 30 sec - exiting
23:30:56 (3504): No heartbeat from core client for 30 sec - exiting
23:30:57 (3504): No heartbeat from core client for 30 sec - exiting
23:30:58 (3504): No heartbeat from core client for 30 sec - exiting
23:30:59 (3504): No heartbeat from core client for 30 sec - exiting
23:31:00 (3504): No heartbeat from core client for 30 sec - exiting
23:31:01 (3504): No heartbeat from core client for 30 sec - exiting
23:31:02 (3504): No heartbeat from core client for 30 sec - exiting
23:31:03 (3504): No heartbeat from core client for 30 sec - exiting
23:31:04 (3504): No heartbeat from core client for 30 sec - exiting
23:31:05 (3504): No heartbeat from core client for 30 sec - exiting
23:31:06 (3504): No heartbeat from core client for 30 sec - exiting
23:31:07 (3504): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0985f4e0 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
03:47:29 (13504): No heartbeat from core client for 30 sec - exiting
03:47:30 (13504): No heartbeat from core client for 30 sec - exiting
03:47:31 (13504): No heartbeat from core client for 30 sec - exiting
03:47:32 (13504): No heartbeat from core client for 30 sec - exiting
03:47:33 (13504): No heartbeat from core client for 30 sec - exiting
03:47:34 (13504): No heartbeat from core client for 30 sec - exiting
03:47:35 (13504): No heartbeat from core client for 30 sec - exiting
03:47:36 (13504): No heartbeat from core client for 30 sec - exiting
09:53:03 (13504): No heartbeat from core client for 30 sec - exiting
09:53:04 (13504): No heartbeat from core client for 30 sec - exiting
09:53:05 (13504): No heartbeat from core client for 30 sec - exiting
09:53:06 (13504): No heartbeat from core client for 30 sec - exiting
09:53:07 (13504): No heartbeat from core client for 30 sec - exiting
09:53:08 (13504): No heartbeat from core client for 30 sec - exiting
09:53:09 (13504): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08a814e0 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
23:54:09 (11133): No heartbeat from core client for 30 sec - exiting
23:54:10 (11133): No heartbeat from core client for 30 sec - exiting
23:54:11 (11133): No heartbeat from core client for 30 sec - exiting
23:54:12 (11133): No heartbeat from core client for 30 sec - exiting
23:54:13 (11133): No heartbeat from core client for 30 sec - exiting
23:54:14 (11133): No heartbeat from core client for 30 sec - exiting
18:58:15 (11133): No heartbeat from core client for 30 sec - exiting
18:58:16 (11133): No heartbeat from core client for 30 sec - exiting
18:58:17 (11133): No heartbeat from core client for 30 sec - exiting
18:58:18 (11133): No heartbeat from core client for 30 sec - exiting
18:58:19 (11133): No heartbeat from core client for 30 sec - exiting
00:10:50 (11133): No heartbeat from core client for 30 sec - exiting
00:10:51 (11133): No heartbeat from core client for 30 sec - exiting
00:10:52 (11133): No heartbeat from core client for 30 sec - exiting
00:10:53 (11133): No heartbeat from core client for 30 sec - exiting
00:10:54 (11133): No heartbeat from core client for 30 sec - exiting
00:10:55 (11133): No heartbeat from core client for 30 sec - exiting
00:59:20 (11133): No heartbeat from core client for 30 sec - exiting
00:59:21 (11133): No heartbeat from core client for 30 sec - exiting
00:59:22 (11133): No heartbeat from core client for 30 sec - exiting
00:59:23 (11133): No heartbeat from core client for 30 sec - exiting
00:59:24 (11133): No heartbeat from core client for 30 sec - exiting
00:59:25 (11133): No heartbeat from core client for 30 sec - exiting
00:59:26 (11133): No heartbeat from core client for 30 sec - exiting
00:59:27 (11133): No heartbeat from core client for 30 sec - exiting
23:58:31 (11133): No heartbeat from core client for 30 sec - exiting
23:58:32 (11133): No heartbeat from core client for 30 sec - exiting
23:58:33 (11133): No heartbeat from core client for 30 sec - exiting
23:58:34 (11133): No heartbeat from core client for 30 sec - exiting
23:58:35 (11133): No heartbeat from core client for 30 sec - exiting
23:58:36 (11133): No heartbeat from core client for 30 sec - exiting
23:58:37 (11133): No heartbeat from core client for 30 sec - exiting
23:58:38 (11133): No heartbeat from core client for 30 sec - exiting
23:58:39 (11133): No heartbeat from core client for 30 sec - exiting
01:01:17 (11133): No heartbeat from core client for 30 sec - exiting
01:01:18 (11133): No heartbeat from core client for 30 sec - exiting
01:01:19 (11133): No heartbeat from core client for 30 sec - exiting
01:01:20 (11133): No heartbeat from core client for 30 sec - exiting
01:01:21 (11133): No heartbeat from core client for 30 sec - exiting
01:01:22 (11133): No heartbeat from core client for 30 sec - exiting
01:01:23 (11133): No heartbeat from core client for 30 sec - exiting
01:01:24 (11133): No heartbeat from core client for 30 sec - exiting
19:52:51 (11133): No heartbeat from core client for 30 sec - exiting
19:52:53 (11133): No heartbeat from core client for 30 sec - exiting
19:52:54 (11133): No heartbeat from core client for 30 sec - exiting
19:52:55 (11133): No heartbeat from core client for 30 sec - exiting
19:52:56 (11133): No heartbeat from core client for 30 sec - exiting
19:52:57 (11133): No heartbeat from core client for 30 sec - exiting
19:52:58 (11133): No heartbeat from core client for 30 sec - exiting
19:52:59 (11133): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0861b4e0 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
20:37:47 (3473): No heartbeat from core client for 30 sec - exiting
20:37:48 (3473): No heartbeat from core client for 30 sec - exiting
20:37:49 (3473): No heartbeat from core client for 30 sec - exiting
20:37:50 (3473): No heartbeat from core client for 30 sec - exiting
20:37:51 (3473): No heartbeat from core client for 30 sec - exiting
20:37:52 (3473): No heartbeat from core client for 30 sec - exiting
20:37:53 (3473): No heartbeat from core client for 30 sec - exiting
19:55:28 (3473): No heartbeat from core client for 30 sec - exiting
19:55:29 (3473): No heartbeat from core client for 30 sec - exiting
19:55:30 (3473): No heartbeat from core client for 30 sec - exiting
19:55:31 (3473): No heartbeat from core client for 30 sec - exiting
19:55:32 (3473): No heartbeat from core client for 30 sec - exiting
19:55:33 (3473): No heartbeat from core client for 30 sec - exiting
19:55:34 (3473): No heartbeat from core client for 30 sec - exiting
19:55:35 (3473): No heartbeat from core client for 30 sec - exiting
00:38:53 (3473): No heartbeat from core client for 30 sec - exiting
00:38:54 (3473): No heartbeat from core client for 30 sec - exiting
00:38:55 (3473): No heartbeat from core client for 30 sec - exiting
00:38:56 (3473): No heartbeat from core client for 30 sec - exiting
00:38:57 (3473): No heartbeat from core client for 30 sec - exiting
00:38:58 (3473): No heartbeat from core client for 30 sec - exiting
00:38:59 (3473): No heartbeat from core client for 30 sec - exiting
19:14:11 (3473): No heartbeat from core client for 30 sec - exiting
19:14:12 (3473): No heartbeat from core client for 30 sec - exiting
19:14:13 (3473): No heartbeat from core client for 30 sec - exiting
19:14:14 (3473): No heartbeat from core client for 30 sec - exiting
19:14:15 (3473): No heartbeat from core client for 30 sec - exiting
19:14:16 (3473): No heartbeat from core client for 30 sec - exiting
19:14:17 (3473): No heartbeat from core client for 30 sec - exiting
19:14:18 (3473): No heartbeat from core client for 30 sec - exiting
19:14:19 (3473): No heartbeat from core client for 30 sec - exiting
09:30:47 (3473): No heartbeat from core client for 30 sec - exiting
09:30:48 (3473): No heartbeat from core client for 30 sec - exiting
09:30:49 (3473): No heartbeat from core client for 30 sec - exiting
09:30:50 (3473): No heartbeat from core client for 30 sec - exiting
09:30:51 (3473): No heartbeat from core client for 30 sec - exiting
09:30:52 (3473): No heartbeat from core client for 30 sec - exiting
09:30:53 (3473): No heartbeat from core client for 30 sec - exiting
09:30:54 (3473): No heartbeat from core client for 30 sec - exiting
09:30:55 (3473): No heartbeat from core client for 30 sec - exiting
09:30:56 (3473): No heartbeat from core client for 30 sec - exiting
09:30:57 (3473): No heartbeat from core client for 30 sec - exiting
09:30:58 (3473): No heartbeat from core client for 30 sec - exiting
09:30:59 (3473): No heartbeat from core client for 30 sec - exiting
09:31:00 (3473): No heartbeat from core client for 30 sec - exiting
09:31:01 (3473): No heartbeat from core client for 30 sec - exiting
00:09:03 (3473): No heartbeat from core client for 30 sec - exiting
00:09:04 (3473): No heartbeat from core client for 30 sec - exiting
00:09:05 (3473): No heartbeat from core client for 30 sec - exiting
00:09:06 (3473): No heartbeat from core client for 30 sec - exiting
00:09:07 (3473): No heartbeat from core client for 30 sec - exiting
00:09:08 (3473): No heartbeat from core client for 30 sec - exiting
00:09:09 (3473): No heartbeat from core client for 30 sec - exiting
00:09:10 (3473): No heartbeat from core client for 30 sec - exiting
01:56:33 (3473): No heartbeat from core client for 30 sec - exiting
01:56:35 (3473): No heartbeat from core client for 30 sec - exiting
01:56:36 (3473): No heartbeat from core client for 30 sec - exiting
01:56:37 (3473): No heartbeat from core client for 30 sec - exiting
01:56:38 (3473): No heartbeat from core client for 30 sec - exiting
01:56:39 (3473): No heartbeat from core client for 30 sec - exiting
01:56:40 (3473): No heartbeat from core client for 30 sec - exiting
01:56:41 (3473): No heartbeat from core client for 30 sec - exiting
21:44:20 (3473): No heartbeat from core client for 30 sec - exiting
21:44:21 (3473): No heartbeat from core client for 30 sec - exiting
21:44:22 (3473): No heartbeat from core client for 30 sec - exiting
21:44:23 (3473): No heartbeat from core client for 30 sec - exiting
21:44:24 (3473): No heartbeat from core client for 30 sec - exiting
21:44:25 (3473): No heartbeat from core client for 30 sec - exiting
21:44:26 (3473): No heartbeat from core client for 30 sec - exiting
21:44:27 (3473): No heartbeat from core client for 30 sec - exiting
20:38:38 (3473): No heartbeat from core client for 30 sec - exiting
20:38:39 (3473): No heartbeat from core client for 30 sec - exiting
20:38:40 (3473): No heartbeat from core client for 30 sec - exiting
20:38:41 (3473): No heartbeat from core client for 30 sec - exiting
20:38:42 (3473): No heartbeat from core client for 30 sec - exiting
20:38:43 (3473): No heartbeat from core client for 30 sec - exiting
20:38:44 (3473): No heartbeat from core client for 30 sec - exiting
20:38:45 (3473): No heartbeat from core client for 30 sec - exiting
20:38:46 (3473): No heartbeat from core client for 30 sec - exiting

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
11 Jul 2013 13:02:24 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 673,920 1,356,415 2.0127
07 Jul 2013 17:19:52 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 518,400 1,056,411 2.0378
07 Jul 2013 01:11:36 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 492,480 1,003,055 2.0367
06 Jul 2013 07:16:54 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 466,560 949,466 2.0350
06 Jul 2013 05:08:17 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 440,640 895,677 2.0327
04 Jul 2013 17:43:02 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 414,720 844,190 2.0356
04 Jul 2013 14:04:39 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 388,800 793,993 2.0422
26 Jun 2013 11:30:52 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 362,880 742,571 2.0463
25 Jun 2013 16:46:19 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 336,960 689,519 2.0463
25 Jun 2013 00:27:04 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 311,040 636,257 2.0456
24 Jun 2013 08:44:52 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 285,120 582,956 2.0446
23 Jun 2013 15:24:35 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 259,200 529,622 2.0433
22 Jun 2013 19:49:13 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 233,280 476,450 2.0424
21 Jun 2013 09:38:50 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 207,360 422,845 2.0392
20 Jun 2013 12:24:34 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 181,440 369,602 2.0370
19 Jun 2013 20:46:05 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 155,520 316,392 2.0344
19 Jun 2013 04:19:26 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 129,600 262,553 2.0259
18 Jun 2013 11:42:01 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 103,680 209,893 2.0244
17 Jun 2013 00:12:03 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 77,760 158,155 2.0339
15 Jun 2013 20:34:36 1284277 15813415 hadcm3n_o1j4_1940_40_008381996_0 51,840 106,511 2.0546


©2024 cpdn.org