climateprediction.net home page
Task 15585935

Task 15585935

Name hadcm3n_483d_1940_40_008300867_0
Workunit 8452002
Created 6 Feb 2013, 13:56:13 UTC
Sent 6 Feb 2013, 13:56:32 UTC
Report deadline 8 May 2013, 21:23:43 UTC
Received 16 Mar 2013, 22:50:55 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1212089
Run time 22 days 6 hours 30 min 22 sec
CPU time 15 days 15 hours 50 min 4 sec
Validate state Invalid
Credit 12,441.60
Device peak FLOPS 1.68 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr 
<core_client_version>7.0.29</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
15:27:36 (8555): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:27:37 (8555): No heartbeat from core client for 30 sec - exiting
15:27:40 (8555): No heartbeat from core client for 30 sec - exiting
15:27:41 (8555): No heartbeat from core client for 30 sec - exiting
15:27:42 (8555): No heartbeat from core client for 30 sec - exiting
15:27:43 (8555): No heartbeat from core client for 30 sec - exiting
03:45:56 (356): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:45:57 (356): No heartbeat from core client for 30 sec - exiting
04:47:14 (16532): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:47:15 (16532): No heartbeat from core client for 30 sec - exiting
04:47:18 (16532): No heartbeat from core client for 30 sec - exiting
04:47:19 (16532): No heartbeat from core client for 30 sec - exiting
04:47:20 (16532): No heartbeat from core client for 30 sec - exiting
04:47:21 (16532): No heartbeat from core client for 30 sec - exiting
04:47:22 (16532): No heartbeat from core client for 30 sec - exiting
05:41:08 (16808): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:16:21 (27538): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:16:22 (27538): No heartbeat from core client for 30 sec - exiting
16:16:23 (27538): No heartbeat from core client for 30 sec - exiting
16:16:24 (27538): No heartbeat from core client for 30 sec - exiting
16:16:28 (27538): No heartbeat from core client for 30 sec - exiting
16:16:29 (27538): No heartbeat from core client for 30 sec - exiting
16:16:30 (27538): No heartbeat from core client for 30 sec - exiting
16:16:31 (27538): No heartbeat from core client for 30 sec - exiting
16:16:32 (27538): No heartbeat from core client for 30 sec - exiting
16:16:34 (27538): No heartbeat from core client for 30 sec - exiting
16:58:36 (11808): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:58:37 (11808): No heartbeat from core client for 30 sec - exiting
16:58:38 (11808): No heartbeat from core client for 30 sec - exiting
16:58:39 (11808): No heartbeat from core client for 30 sec - exiting
16:58:40 (11808): No heartbeat from core client for 30 sec - exiting
17:23:58 (6197): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:22:06 (4983): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
22:03:55 (4990): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
22:03:57 (4990): No heartbeat from core client for 30 sec - exiting
22:03:58 (4990): No heartbeat from core client for 30 sec - exiting
22:03:59 (4990): No heartbeat from core client for 30 sec - exiting
22:04:00 (4990): No heartbeat from core client for 30 sec - exiting
22:04:01 (4990): No heartbeat from core client for 30 sec - exiting
22:04:06 (4990): No heartbeat from core client for 30 sec - exiting
22:04:07 (4990): No heartbeat from core client for 30 sec - exiting
22:04:08 (4990): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
00:20:15 (4922): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
00:34:25 (5550): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
00:34:26 (5550): No heartbeat from core client for 30 sec - exiting
00:34:27 (5550): No heartbeat from core client for 30 sec - exiting
00:34:29 (5550): No heartbeat from core client for 30 sec - exiting
00:34:30 (5550): No heartbeat from core client for 30 sec - exiting
00:34:31 (5550): No heartbeat from core client for 30 sec - exiting
00:34:32 (5550): No heartbeat from core client for 30 sec - exiting
00:34:35 (5550): No heartbeat from core client for 30 sec - exiting
00:34:36 (5550): No heartbeat from core client for 30 sec - exiting
00:34:37 (5550): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
01:59:29 (5647): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:59:30 (5647): No heartbeat from core client for 30 sec - exiting
01:59:31 (5647): No heartbeat from core client for 30 sec - exiting
01:59:32 (5647): No heartbeat from core client for 30 sec - exiting
01:59:33 (5647): No heartbeat from core client for 30 sec - exiting
01:59:34 (5647): No heartbeat from core client for 30 sec - exiting
04:07:48 (6208): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:49:46 (8202): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:17:26 (9119): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:17:27 (9119): No heartbeat from core client for 30 sec - exiting
07:17:28 (9119): No heartbeat from core client for 30 sec - exiting
07:17:29 (9119): No heartbeat from core client for 30 sec - exiting
07:17:30 (9119): No heartbeat from core client for 30 sec - exiting
07:17:31 (9119): No heartbeat from core client for 30 sec - exiting
07:17:33 (9119): No heartbeat from core client for 30 sec - exiting
07:17:34 (9119): No heartbeat from core client for 30 sec - exiting
07:17:35 (9119): No heartbeat from core client for 30 sec - exiting
07:17:36 (9119): No heartbeat from core client for 30 sec - exiting
07:17:39 (9119): No heartbeat from core client for 30 sec - exiting
07:17:40 (9119): No heartbeat from core client for 30 sec - exiting
07:17:41 (9119): No heartbeat from core client for 30 sec - exiting
07:17:42 (9119): No heartbeat from core client for 30 sec - exiting
07:17:44 (9119): No heartbeat from core client for 30 sec - exiting
07:17:45 (9119): No heartbeat from core client for 30 sec - exiting
07:17:46 (9119): No heartbeat from core client for 30 sec - exiting
07:17:47 (9119): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
09:37:39 (9256): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:05:25 (10235): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:05:27 (10235): No heartbeat from core client for 30 sec - exiting
11:05:28 (10235): No heartbeat from core client for 30 sec - exiting
11:05:29 (10235): No heartbeat from core client for 30 sec - exiting
11:05:30 (10235): No heartbeat from core client for 30 sec - exiting
11:05:31 (10235): No heartbeat from core client for 30 sec - exiting
11:05:32 (10235): No heartbeat from core client for 30 sec - exiting
11:05:33 (10235): No heartbeat from core client for 30 sec - exiting
11:05:34 (10235): No heartbeat from core client for 30 sec - exiting
12:51:08 (10644): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:01:01 (11168): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:01:04 (11168): No heartbeat from core client for 30 sec - exiting
21:01:05 (11168): No heartbeat from core client for 30 sec - exiting
21:01:06 (11168): No heartbeat from core client for 30 sec - exiting
21:01:07 (11168): No heartbeat from core client for 30 sec - exiting
21:01:08 (11168): No heartbeat from core client for 30 sec - exiting
21:10:00 (16862): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:10:01 (16862): No heartbeat from core client for 30 sec - exiting
21:10:02 (16862): No heartbeat from core client for 30 sec - exiting
21:10:04 (16862): No heartbeat from core client for 30 sec - exiting
21:10:05 (16862): No heartbeat from core client for 30 sec - exiting
21:10:06 (16862): No heartbeat from core client for 30 sec - exiting
21:10:07 (16862): No heartbeat from core client for 30 sec - exiting
21:10:08 (16862): No heartbeat from core client for 30 sec - exiting
21:10:09 (16862): No heartbeat from core client for 30 sec - exiting
21:10:10 (16862): No heartbeat from core client for 30 sec - exiting
21:10:11 (16862): No heartbeat from core client for 30 sec - exiting
21:10:12 (16862): No heartbeat from core client for 30 sec - exiting
21:10:16 (16862): No heartbeat from core client for 30 sec - exiting
21:10:17 (16862): No heartbeat from core client for 30 sec - exiting
21:10:18 (16862): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
21:15:31 (16911): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:20:38 (16937): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:20:39 (16937): No heartbeat from core client for 30 sec - exiting
21:20:40 (16937): No heartbeat from core client for 30 sec - exiting
21:20:41 (16937): No heartbeat from core client for 30 sec - exiting
21:20:43 (16937): No heartbeat from core client for 30 sec - exiting
21:20:44 (16937): No heartbeat from core client for 30 sec - exiting
21:20:45 (16937): No heartbeat from core client for 30 sec - exiting
21:20:46 (16937): No heartbeat from core client for 30 sec - exiting
21:20:47 (16937): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
00:12:12 (16973): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
00:12:13 (16973): No heartbeat from core client for 30 sec - exiting
00:56:27 (17943): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
00:56:29 (17943): No heartbeat from core client for 30 sec - exiting
00:56:30 (17943): No heartbeat from core client for 30 sec - exiting
00:56:31 (17943): No heartbeat from core client for 30 sec - exiting
00:56:32 (17943): No heartbeat from core client for 30 sec - exiting
00:56:33 (17943): No heartbeat from core client for 30 sec - exiting
00:56:34 (17943): No heartbeat from core client for 30 sec - exiting
17:55:18 (18152): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:55:20 (18152): No heartbeat from core client for 30 sec - exiting
17:55:21 (18152): No heartbeat from core client for 30 sec - exiting
17:55:22 (18152): No heartbeat from core client for 30 sec - exiting
17:55:23 (18152): No heartbeat from core client for 30 sec - exiting
17:55:24 (18152): No heartbeat from core client for 30 sec - exiting
17:55:25 (18152): No heartbeat from core client for 30 sec - exiting
17:55:26 (18152): No heartbeat from core client for 30 sec - exiting
17:55:28 (18152): No heartbeat from core client for 30 sec - exiting
17:55:29 (18152): No heartbeat from core client for 30 sec - exiting
03:47:21 (20244): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:39:02 (24775): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:39:03 (24775): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x097d6d98 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09814460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09c59460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x099b1460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0a011460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08b33460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x082c5460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x096fe460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09233460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08cf6460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
05:30:26 (4987): No heartbeat from core client for 30 sec - exiting
05:30:27 (4987): No heartbeat from core client for 30 sec - exiting
05:30:28 (4987): No heartbeat from core client for 30 sec - exiting
05:30:30 (4987): No heartbeat from core client for 30 sec - exiting
05:30:31 (4987): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x095a5460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
05:41:15 (4796): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08f82460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
00:16:58 (4954): No heartbeat from core client for 30 sec - exiting
00:16:59 (4954): No heartbeat from core client for 30 sec - exiting
00:17:00 (4954): No heartbeat from core client for 30 sec - exiting
00:17:01 (4954): No heartbeat from core client for 30 sec - exiting
00:17:02 (4954): No heartbeat from core client for 30 sec - exiting
00:17:07 (4954): No heartbeat from core client for 30 sec - exiting
00:17:08 (4954): No heartbeat from core client for 30 sec - exiting
00:17:09 (4954): No heartbeat from core client for 30 sec - exiting
00:17:10 (4954): No heartbeat from core client for 30 sec - exiting
00:17:11 (4954): No heartbeat from core client for 30 sec - exiting
00:17:12 (4954): No heartbeat from core client for 30 sec - exiting
00:17:15 (4954): No heartbeat from core client for 30 sec - exiting
00:17:16 (4954): No heartbeat from core client for 30 sec - exiting
00:17:17 (4954): No heartbeat from core client for 30 sec - exiting
00:17:18 (4954): No heartbeat from core client for 30 sec - exiting
00:17:19 (4954): No heartbeat from core client for 30 sec - exiting
00:17:20 (4954): No heartbeat from core client for 30 sec - exiting
00:17:32 (4954): No heartbeat from core client for 30 sec - exiting
00:17:33 (4954): No heartbeat from core client for 30 sec - exiting
00:17:34 (4954): No heartbeat from core client for 30 sec - exiting
00:26:38 (4954): No heartbeat from core client for 30 sec - exiting
00:26:39 (4954): No heartbeat from core client for 30 sec - exiting
00:26:43 (4954): No heartbeat from core client for 30 sec - exiting
00:26:44 (4954): No heartbeat from core client for 30 sec - exiting
00:26:45 (4954): No heartbeat from core client for 30 sec - exiting
00:26:46 (4954): No heartbeat from core client for 30 sec - exiting
00:26:49 (4954): No heartbeat from core client for 30 sec - exiting
00:26:50 (4954): No heartbeat from core client for 30 sec - exiting
00:26:51 (4954): No heartbeat from core client for 30 sec - exiting
00:26:52 (4954): No heartbeat from core client for 30 sec - exiting
00:26:55 (4954): No heartbeat from core client for 30 sec - exiting
00:26:56 (4954): No heartbeat from core client for 30 sec - exiting
00:26:57 (4954): No heartbeat from core client for 30 sec - exiting
00:26:58 (4954): No heartbeat from core client for 30 sec - exiting
00:26:59 (4954): No heartbeat from core client for 30 sec - exiting
00:27:02 (4954): No heartbeat from core client for 30 sec - exiting
00:27:03 (4954): No heartbeat from core client for 30 sec - exiting
00:27:04 (4954): No heartbeat from core client for 30 sec - exiting
00:27:05 (4954): No heartbeat from core client for 30 sec - exiting
00:27:07 (4954): No heartbeat from core client for 30 sec - exiting
00:27:08 (4954): No heartbeat from core client for 30 sec - exiting
00:27:09 (4954): No heartbeat from core client for 30 sec - exiting
00:27:10 (4954): No heartbeat from core client for 30 sec - exiting
00:27:13 (4954): No heartbeat from core client for 30 sec - exiting
00:27:14 (4954): No heartbeat from core client for 30 sec - exiting
00:27:15 (4954): No heartbeat from core client for 30 sec - exiting
00:27:16 (4954): No heartbeat from core client for 30 sec - exiting
00:27:20 (4954): No heartbeat from core client for 30 sec - exiting
00:27:21 (4954): No heartbeat from core client for 30 sec - exiting
00:27:22 (4954): No heartbeat from core client for 30 sec - exiting
00:27:23 (4954): No heartbeat from core client for 30 sec - exiting
00:27:24 (4954): No heartbeat from core client for 30 sec - exiting
03:23:37 (4954): No heartbeat from core client for 30 sec - exiting
03:23:39 (4954): No heartbeat from core client for 30 sec - exiting
03:23:40 (4954): No heartbeat from core client for 30 sec - exiting
03:23:41 (4954): No heartbeat from core client for 30 sec - exiting
03:23:42 (4954): No heartbeat from core client for 30 sec - exiting
03:23:43 (4954): No heartbeat from core client for 30 sec - exiting
03:23:44 (4954): No heartbeat from core client for 30 sec - exiting
10:04:49 (4954): No heartbeat from core client for 30 sec - exiting
10:04:50 (4954): No heartbeat from core client for 30 sec - exiting
10:04:51 (4954): No heartbeat from core client for 30 sec - exiting
10:04:53 (4954): No heartbeat from core client for 30 sec - exiting
10:04:54 (4954): No heartbeat from core client for 30 sec - exiting
10:04:55 (4954): No heartbeat from core client for 30 sec - exiting
10:04:56 (4954): No heartbeat from core client for 30 sec - exiting
10:04:57 (4954): No heartbeat from core client for 30 sec - exiting
10:04:59 (4954): No heartbeat from core client for 30 sec - exiting
10:05:00 (4954): No heartbeat from core client for 30 sec - exiting
10:05:01 (4954): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08d3b460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
00:20:07 (29318): No heartbeat from core client for 30 sec - exiting
00:20:10 (29318): No heartbeat from core client for 30 sec - exiting
00:20:11 (29318): No heartbeat from core client for 30 sec - exiting
00:20:12 (29318): No heartbeat from core client for 30 sec - exiting
00:20:13 (29318): No heartbeat from core client for 30 sec - exiting
00:20:14 (29318): No heartbeat from core client for 30 sec - exiting
00:20:15 (29318): No heartbeat from core client for 30 sec - exiting
00:20:17 (29318): No heartbeat from core client for 30 sec - exiting
00:20:18 (29318): No heartbeat from core client for 30 sec - exiting
00:20:19 (29318): No heartbeat from core client for 30 sec - exiting
12:22:29 (29318): No heartbeat from core client for 30 sec - exiting
12:22:30 (29318): No heartbeat from core client for 30 sec - exiting
12:22:31 (29318): No heartbeat from core client for 30 sec - exiting
12:22:32 (29318): No heartbeat from core client for 30 sec - exiting
12:22:33 (29318): No heartbeat from core client for 30 sec - exiting
12:22:34 (29318): No heartbeat from core client for 30 sec - exiting
12:22:35 (29318): No heartbeat from core client for 30 sec - exiting
12:22:37 (29318): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x082fa460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
17:05:11 (5512): No heartbeat from core client for 30 sec - exiting
19:49:58 (5512): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0877a460 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
00:57:59 (19693): No heartbeat from core client for 30 sec - exiting
00:58:00 (19693): No heartbeat from core client for 30 sec - exiting
00:58:01 (19693): No heartbeat from core client for 30 sec - exiting
00:58:02 (19693): No heartbeat from core client for 30 sec - exiting
00:58:05 (19693): No heartbeat from core client for 30 sec - exiting
00:58:06 (19693): No heartbeat from core client for 30 sec - exiting
00:58:07 (19693): No heartbeat from core client for 30 sec - exiting
00:58:08 (19693): No heartbeat from core client for 30 sec - exiting
00:58:11 (19693): No heartbeat from core client for 30 sec - exiting
00:58:12 (19693): No heartbeat from core client for 30 sec - exiting
00:58:13 (19693): No heartbeat from core client for 30 sec - exiting
00:58:14 (19693): No heartbeat from core client for 30 sec - exiting
00:58:18 (19693): No heartbeat from core client for 30 sec - exiting
00:58:19 (19693): No heartbeat from core client for 30 sec - exiting
00:58:20 (19693): No heartbeat from core client for 30 sec - exiting
00:58:21 (19693): No heartbeat from core client for 30 sec - exiting
00:58:22 (19693): No heartbeat from core client for 30 sec - exiting
01:18:33 (19693): No heartbeat from core client for 30 sec - exiting
01:18:34 (19693): No heartbeat from core client for 30 sec - exiting
01:18:35 (19693): No heartbeat from core client for 30 sec - exiting
01:18:36 (19693): No heartbeat from core client for 30 sec - exiting
01:18:37 (19693): No heartbeat from core client for 30 sec - exiting
01:18:38 (19693): No heartbeat from core client for 30 sec - exiting
01:18:40 (19693): No heartbeat from core client for 30 sec - exiting
01:18:41 (19693): No heartbeat from core client for 30 sec - exiting
01:18:42 (19693): No heartbeat from core client for 30 sec - exiting
01:18:43 (19693): No heartbeat from core client for 30 sec - exiting
01:18:46 (19693): No heartbeat from core client for 30 sec - exiting
01:18:47 (19693): No heartbeat from core client for 30 sec - exiting
01:18:48 (19693): No heartbeat from core client for 30 sec - exiting
01:18:49 (19693): No heartbeat from core client for 30 sec - exiting
01:18:52 (19693): No heartbeat from core client for 30 sec - exiting
01:18:53 (19693): No heartbeat from core client for 30 sec - exiting
01:18:54 (19693): No heartbeat from core client for 30 sec - exiting
01:18:55 (19693): No heartbeat from core client for 30 sec - exiting
01:18:58 (19693): No heartbeat from core client for 30 sec - exiting
01:18:59 (19693): No heartbeat from core client for 30 sec - exiting
01:19:00 (19693): No heartbeat from core client for 30 sec - exiting
01:19:01 (19693): No heartbeat from core client for 30 sec - exiting
01:19:03 (19693): No heartbeat from core client for 30 sec - exiting
01:19:04 (19693): No heartbeat from core client for 30 sec - exiting
01:19:05 (19693): No heartbeat from core client for 30 sec - exiting
01:19:06 (19693): No heartbeat from core client for 30 sec - exiting
10:13:35 (19693): No heartbeat from core client for 30 sec - exiting
10:13:36 (19693): No heartbeat from core client for 30 sec - exiting
00:56:26 (19693): No heartbeat from core client for 30 sec - exiting
00:56:27 (19693): No heartbeat from core client for 30 sec - exiting
00:56:28 (19693): No heartbeat from core client for 30 sec - exiting
00:56:29 (19693): No heartbeat from core client for 30 sec - exiting
00:56:31 (19693): No heartbeat from core client for 30 sec - exiting
00:56:32 (19693): No heartbeat from core client for 30 sec - exiting
00:56:33 (19693): No heartbeat from core client for 30 sec - exiting
00:56:34 (19693): No heartbeat from core client for 30 sec - exiting
22:07:54 (19693): No heartbeat from core client for 30 sec - exiting
execv: No such file or directory

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
01 Mar 2013 01:20:40 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 1,036,800 1,352,455 1.3045
28 Feb 2013 11:33:24 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 1,010,880 1,317,295 1.3031
27 Feb 2013 21:54:27 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 984,960 1,282,594 1.3022
27 Feb 2013 08:26:44 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 959,040 1,247,946 1.3012
26 Feb 2013 18:41:59 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 933,120 1,213,453 1.3004
26 Feb 2013 04:54:05 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 907,200 1,178,614 1.2992
25 Feb 2013 14:56:46 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 881,280 1,143,198 1.2972
25 Feb 2013 01:06:50 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 855,360 1,108,465 1.2959
24 Feb 2013 09:10:33 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 829,440 1,075,652 1.2968
23 Feb 2013 17:41:39 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 803,520 1,041,923 1.2967
23 Feb 2013 04:01:08 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 777,600 1,007,084 1.2951
22 Feb 2013 14:25:23 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 751,680 972,036 1.2932
22 Feb 2013 00:30:13 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 725,760 936,959 1.2910
21 Feb 2013 10:50:06 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 699,840 901,956 1.2888
20 Feb 2013 21:16:26 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 673,920 867,147 1.2867
20 Feb 2013 13:40:18 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 648,000 832,318 1.2844
20 Feb 2013 13:40:18 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 622,080 800,336 1.2865
20 Feb 2013 13:40:18 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 596,160 769,533 1.2908
20 Feb 2013 13:40:18 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 570,240 738,710 1.2954
18 Feb 2013 05:22:34 1212089 15585935 hadcm3n_483d_1940_40_008300867_0 544,320 705,754 1.2966


©2024 cpdn.org