Task 13626897

Name	hadcm3n_o1dr_1940_40_007543473_1
Workunit	7740705
Created	10 Nov 2011, 1:45:51 UTC
Sent	16 Nov 2011, 8:02:40 UTC
Report deadline	15 Feb 2012, 15:29:51 UTC
Received	18 Jan 2012, 14:09:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1072022
Run time	20 days 12 hours 28 min 33 sec
CPU time	16 days 19 hours 43 min 53 sec
Validate state	Invalid
Credit	8,398.08
Device peak FLOPS	2.89 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.59</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 09:36:12 (2639): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:54:36 (5675): No heartbeat from core client for 30 sec - exiting 09:54:39 (5675): No heartbeat from core client for 30 sec - exiting 09:54:40 (5675): No heartbeat from core client for 30 sec - exiting 09:54:41 (5675): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:54:42 (5675): No heartbeat from core client for 30 sec - exiting 09:54:43 (5675): No heartbeat from core client for 30 sec - exiting 09:54:44 (5675): No heartbeat from core client for 30 sec - exiting 09:54:45 (5675): No heartbeat from core client for 30 sec - exiting 10:41:14 (7358): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:57:27 (13067): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:22:46 (14695): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:22:47 (14695): No heartbeat from core client for 30 sec - exiting 11:22:48 (14695): No heartbeat from core client for 30 sec - exiting 11:22:49 (14695): No heartbeat from core client for 30 sec - exiting 11:22:50 (14695): No heartbeat from core client for 30 sec - exiting 11:22:51 (14695): No heartbeat from core client for 30 sec - exiting 11:22:52 (14695): No heartbeat from core client for 30 sec - exiting 11:22:53 (14695): No heartbeat from core client for 30 sec - exiting 12:59:06 (17404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:03:13 (28560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:04:22 (28560): No heartbeat from core client for 30 sec - exiting 13:04:23 (28560): No heartbeat from core client for 30 sec - exiting 13:04:24 (28560): No heartbeat from core client for 30 sec - exiting 13:12:17 (29109): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:23:58 (29942): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:08:58 (31040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:14:03 (3670): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:30:30 (2407): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:32:37 (10013): No heartbeat from core client for 30 sec - exiting 14:32:54 (10013): No heartbeat from core client for 30 sec - exiting 14:36:31 (10013): No heartbeat from core client for 30 sec - exiting 14:37:02 (10013): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:12:03 (11404): No heartbeat from core client for 30 sec - exiting 16:15:12 (11404): No heartbeat from core client for 30 sec - exiting 16:15:13 (11404): No heartbeat from core client for 30 sec - exiting 16:15:14 (11404): No heartbeat from core client for 30 sec - exiting 16:15:15 (11404): No heartbeat from core client for 30 sec - exiting 16:15:17 (11404): No heartbeat from core client for 30 sec - exiting 16:15:18 (11404): No heartbeat from core client for 30 sec - exiting 16:15:19 (11404): No heartbeat from core client for 30 sec - exiting 16:15:20 (11404): No heartbeat from core client for 30 sec - exiting 16:15:21 (11404): No heartbeat from core client for 30 sec - exiting 16:15:22 (11404): No heartbeat from core client for 30 sec - exiting 16:15:23 (11404): No heartbeat from core client for 30 sec - exiting 16:15:24 (11404): No heartbeat from core client for 30 sec - exiting 16:15:25 (11404): No heartbeat from core client for 30 sec - exiting 16:15:27 (11404): No heartbeat from core client for 30 sec - exiting 16:15:28 (11404): No heartbeat from core client for 30 sec - exiting 16:15:29 (11404): No heartbeat from core client for 30 sec - exiting 16:15:31 (11404): No heartbeat from core client for 30 sec - exiting 16:15:32 (11404): No heartbeat from core client for 30 sec - exiting 16:15:33 (11404): No heartbeat from core client for 30 sec - exiting 16:15:34 (11404): No heartbeat from core client for 30 sec - exiting 16:15:35 (11404): No heartbeat from core client for 30 sec - exiting 16:15:36 (11404): No heartbeat from core client for 30 sec - exiting 16:15:37 (11404): No heartbeat from core client for 30 sec - exiting 16:15:38 (11404): No heartbeat from core client for 30 sec - exiting 16:15:39 (11404): No heartbeat from core client for 30 sec - exiting 16:15:40 (11404): No heartbeat from core client for 30 sec - exiting 16:15:41 (11404): No heartbeat from core client for 30 sec - exiting 16:15:42 (11404): No heartbeat from core client for 30 sec - exiting 16:15:43 (11404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:15:44 (11404): No heartbeat from core client for 30 sec - exiting 16:15:45 (11404): No heartbeat from core client for 30 sec - exiting 16:15:46 (11404): No heartbeat from core client for 30 sec - exiting 16:15:47 (11404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:57:51 (2615): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:57:53 (2615): No heartbeat from core client for 30 sec - exiting 16:19:19 (25587): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:25:12 (25587): No heartbeat from core client for 30 sec - exiting 16:25:13 (25587): No heartbeat from core client for 30 sec - exiting 16:25:14 (25587): No heartbeat from core client for 30 sec - exiting 16:25:15 (25587): No heartbeat from core client for 30 sec - exiting 16:25:16 (25587): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:12:16 (3162): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:12:53 (3162): No heartbeat from core client for 30 sec - exiting 11:12:54 (3162): No heartbeat from core client for 30 sec - exiting 11:12:55 (3162): No heartbeat from core client for 30 sec - exiting 13:55:41 (18655): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:08:28 (3415): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:10:50 (3415): No heartbeat from core client for 30 sec - exiting 19:10:51 (3415): No heartbeat from core client for 30 sec - exiting 19:10:52 (3415): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:01:11 (3218): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:46:12 (2950): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:46:14 (2950): No heartbeat from core client for 30 sec - exiting 20:39:18 (29282): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:39:20 (29282): No heartbeat from core client for 30 sec - exiting 20:39:21 (29282): No heartbeat from core client for 30 sec - exiting 20:39:22 (29282): No heartbeat from core client for 30 sec - exiting 20:39:23 (29282): No heartbeat from core client for 30 sec - exiting 20:39:24 (29282): No heartbeat from core client for 30 sec - exiting 20:39:25 (29282): No heartbeat from core client for 30 sec - exiting 20:39:26 (29282): No heartbeat from core client for 30 sec - exiting 20:39:27 (29282): No heartbeat from core client for 30 sec - exiting 20:39:28 (29282): No heartbeat from core client for 30 sec - exiting 20:39:29 (29282): No heartbeat from core client for 30 sec - exiting 20:39:30 (29282): No heartbeat from core client for 30 sec - exiting 20:39:31 (29282): No heartbeat from core client for 30 sec - exiting 20:39:32 (29282): No heartbeat from core client for 30 sec - exiting 20:39:33 (29282): No heartbeat from core client for 30 sec - exiting 20:39:34 (29282): No heartbeat from core client for 30 sec - exiting 20:39:35 (29282): No heartbeat from core client for 30 sec - exiting 20:39:36 (29282): No heartbeat from core client for 30 sec - exiting 20:39:37 (29282): No heartbeat from core client for 30 sec - exiting 20:39:38 (29282): No heartbeat from core client for 30 sec - exiting 20:39:39 (29282): No heartbeat from core client for 30 sec - exiting 20:39:41 (29282): No heartbeat from core client for 30 sec - exiting 20:39:42 (29282): No heartbeat from core client for 30 sec - exiting 20:39:43 (29282): No heartbeat from core client for 30 sec - exiting 21:25:36 (30866): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:30:05 (11661): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:30:06 (11661): No heartbeat from core client for 30 sec - exiting 21:30:07 (11661): No heartbeat from core client for 30 sec - exiting 21:30:08 (11661): No heartbeat from core client for 30 sec - exiting 21:30:09 (11661): No heartbeat from core client for 30 sec - exiting 21:30:10 (11661): No heartbeat from core client for 30 sec - exiting 21:30:11 (11661): No heartbeat from core client for 30 sec - exiting 21:30:12 (11661): No heartbeat from core client for 30 sec - exiting 21:30:13 (11661): No heartbeat from core client for 30 sec - exiting 21:30:14 (11661): No heartbeat from core client for 30 sec - exiting 21:30:15 (11661): No heartbeat from core client for 30 sec - exiting 21:30:16 (11661): No heartbeat from core client for 30 sec - exiting 21:30:17 (11661): No heartbeat from core client for 30 sec - exiting 21:30:18 (11661): No heartbeat from core client for 30 sec - exiting 21:36:32 (13433): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:36:33 (13433): No heartbeat from core client for 30 sec - exiting 21:36:34 (13433): No heartbeat from core client for 30 sec - exiting 21:36:35 (13433): No heartbeat from core client for 30 sec - exiting 21:36:36 (13433): No heartbeat from core client for 30 sec - exiting 21:36:37 (13433): No heartbeat from core client for 30 sec - exiting 21:36:38 (13433): No heartbeat from core client for 30 sec - exiting 21:36:39 (13433): No heartbeat from core client for 30 sec - exiting 21:36:40 (13433): No heartbeat from core client for 30 sec - exiting 21:36:41 (13433): No heartbeat from core client for 30 sec - exiting 21:36:42 (13433): No heartbeat from core client for 30 sec - exiting 21:36:43 (13433): No heartbeat from core client for 30 sec - exiting 21:36:44 (13433): No heartbeat from core client for 30 sec - exiting 21:36:45 (13433): No heartbeat from core client for 30 sec - exiting 21:36:46 (13433): No heartbeat from core client for 30 sec - exiting 21:36:47 (13433): No heartbeat from core client for 30 sec - exiting 21:36:48 (13433): No heartbeat from core client for 30 sec - exiting 21:36:49 (13433): No heartbeat from core client for 30 sec - exiting 21:36:50 (13433): No heartbeat from core client for 30 sec - exiting 21:36:51 (13433): No heartbeat from core client for 30 sec - exiting 21:41:05 (15074): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:42:25 (16704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:42:28 (16704): No heartbeat from core client for 30 sec - exiting 22:42:29 (16704): No heartbeat from core client for 30 sec - exiting 22:42:30 (16704): No heartbeat from core client for 30 sec - exiting 22:42:31 (16704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:01:19 (2846): No heartbeat from core client for 30 sec - exiting 17:01:25 (2846): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:01:36 (2846): No heartbeat from core client for 30 sec - exiting 17:01:40 (2846): No heartbeat from core client for 30 sec - exiting 17:01:41 (2846): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 02:46:36 (3754): No heartbeat from core client for 30 sec - exiting 02:46:39 (3754): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:46:40 (3754): No heartbeat from core client for 30 sec - exiting 02:46:41 (3754): No heartbeat from core client for 30 sec - exiting 02:46:42 (3754): No heartbeat from core client for 30 sec - exiting 02:46:43 (3754): No heartbeat from core client for 30 sec - exiting 02:46:44 (3754): No heartbeat from core client for 30 sec - exiting 02:46:45 (3754): No heartbeat from core client for 30 sec - exiting 02:46:46 (3754): No heartbeat from core client for 30 sec - exiting 02:46:47 (3754): No heartbeat from core client for 30 sec - exiting 02:46:48 (3754): No heartbeat from core client for 30 sec - exiting 02:46:49 (3754): No heartbeat from core client for 30 sec - exiting 02:46:50 (3754): No heartbeat from core client for 30 sec - exiting 02:46:51 (3754): No heartbeat from core client for 30 sec - exiting 02:46:52 (3754): No heartbeat from core client for 30 sec - exiting 02:46:53 (3754): No heartbeat from core client for 30 sec - exiting 02:46:54 (3754): No heartbeat from core client for 30 sec - exiting 02:46:55 (3754): No heartbeat from core client for 30 sec - exiting 02:46:56 (3754): No heartbeat from core client for 30 sec - exiting 02:46:57 (3754): No heartbeat from core client for 30 sec - exiting 02:46:58 (3754): No heartbeat from core client for 30 sec - exiting 02:46:59 (3754): No heartbeat from core client for 30 sec - exiting 02:47:00 (3754): No heartbeat from core client for 30 sec - exiting 02:47:01 (3754): No heartbeat from core client for 30 sec - exiting 02:47:02 (3754): No heartbeat from core client for 30 sec - exiting 02:47:03 (3754): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:31:52 (3221): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:47:57 (3333): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:55:45 (28249): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:58 (31336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:23:42 (4183): No heartbeat from core client for 30 sec - exiting 10:23:44 (4183): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:00 (12234): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:58:54 (3384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:59:28 (3384): No heartbeat from core client for 30 sec - exiting 15:59:29 (3384): No heartbeat from core client for 30 sec - exiting 15:59:30 (3384): No heartbeat from core client for 30 sec - exiting 15:59:31 (3384): No heartbeat from core client for 30 sec - exiting 15:59:32 (3384): No heartbeat from core client for 30 sec - exiting 15:59:33 (3384): No heartbeat from core client for 30 sec - exiting 15:59:34 (3384): No heartbeat from core client for 30 sec - exiting 15:59:35 (3384): No heartbeat from core client for 30 sec - exiting 15:59:36 (3384): No heartbeat from core client for 30 sec - exiting 15:59:37 (3384): No heartbeat from core client for 30 sec - exiting 15:59:38 (3384): No heartbeat from core client for 30 sec - exiting 15:59:39 (3384): No heartbeat from core client for 30 sec - exiting 15:59:40 (3384): No heartbeat from core client for 30 sec - exiting 15:59:41 (3384): No heartbeat from core client for 30 sec - exiting 15:59:42 (3384): No heartbeat from core client for 30 sec - exiting 15:59:43 (3384): No heartbeat from core client for 30 sec - exiting 15:59:44 (3384): No heartbeat from core client for 30 sec - exiting 15:59:45 (3384): No heartbeat from core client for 30 sec - exiting 15:59:46 (3384): No heartbeat from core client for 30 sec - exiting 15:59:47 (3384): No heartbeat from core client for 30 sec - exiting 15:59:48 (3384): No heartbeat from core client for 30 sec - exiting 15:59:49 (3384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 09:19:50 (3389): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:25:32 (3861): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:58:44 (3694): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:25:16 (3940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 17:01:36 (27221): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:02:23 (29413): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:04:02 (27348): No heartbeat from core client for 30 sec - exiting 16:04:03 (27348): No heartbeat from core client for 30 sec - exiting 16:04:04 (27348): No heartbeat from core client for 30 sec - exiting 16:04:05 (27348): No heartbeat from core client for 30 sec - exiting 16:04:06 (27348): No heartbeat from core client for 30 sec - exiting 16:04:07 (27348): No heartbeat from core client for 30 sec - exiting 16:04:08 (27348): No heartbeat from core client for 30 sec - exiting 16:04:09 (27348): No heartbeat from core client for 30 sec - exiting 16:04:10 (27348): No heartbeat from core client for 30 sec - exiting 16:04:11 (27348): No heartbeat from core client for 30 sec - exiting 16:04:12 (27348): No heartbeat from core client for 30 sec - exiting 16:04:13 (27348): No heartbeat from core client for 30 sec - exiting 16:04:14 (27348): No heartbeat from core client for 30 sec - exiting 16:04:15 (27348): No heartbeat from core client for 30 sec - exiting 16:04:16 (27348): No heartbeat from core client for 30 sec - exiting 16:04:17 (27348): No heartbeat from core client for 30 sec - exiting 16:04:18 (27348): No heartbeat from core client for 30 sec - exiting 16:04:19 (27348): No heartbeat from core client for 30 sec - exiting 16:04:20 (27348): No heartbeat from core client for 30 sec - exiting 16:04:21 (27348): No heartbeat from core client for 30 sec - exiting 16:04:22 (27348): No heartbeat from core client for 30 sec - exiting 16:04:23 (27348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:02:09 (4774): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:16 (4774): No heartbeat from core client for 30 sec - exiting 18:02:17 (4774): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:04:57 (3808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:32:43 (3965): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:29:10 (4083): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:33:51 (4083): No heartbeat from core client for 30 sec - exiting 12:33:52 (4083): No heartbeat from core client for 30 sec - exiting 12:33:53 (4083): No heartbeat from core client for 30 sec - exiting 12:33:54 (4083): No heartbeat from core client for 30 sec - exiting 12:33:55 (4083): No heartbeat from core client for 30 sec - exiting 12:33:56 (4083): No heartbeat from core client for 30 sec - exiting 12:33:57 (4083): No heartbeat from core client for 30 sec - exiting 12:33:58 (4083): No heartbeat from core client for 30 sec - exiting 12:34:02 (4083): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Signal 3 received, exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3812, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:58:28 (3884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:43:08 (3995): No heartbeat from core client for 30 sec - exiting 14:43:26 (3995): No heartbeat from core client for 30 sec - exiting 14:43:27 (3995): No heartbeat from core client for 30 sec - exiting 14:43:28 (3995): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:43:29 (3995): No heartbeat from core client for 30 sec - exiting 14:43:30 (3995): No heartbeat from core client for 30 sec - exiting 14:43:31 (3995): No heartbeat from core client for 30 sec - exiting 14:43:32 (3995): No heartbeat from core client for 30 sec - exiting 14:43:34 (3995): No heartbeat from core client for 30 sec - exiting 14:43:35 (3995): No heartbeat from core client for 30 sec - exiting 14:43:36 (3995): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf776c400] [0xf776c430] /lib32/libc.so.6(gsignal+0x51)[0xf75d1ea1] /lib32/libc.so.6(abort+0x17e)[0xf75d52ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf75bde37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31607, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7733400] [0xf7733430] /lib32/libc.so.6(gsignal+0x51)[0xf7598ea1] /lib32/libc.so.6(abort+0x17e)[0xf759c2ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf7584e37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31607, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf773e400] [0xf773e430] /lib32/libc.so.6(gsignal+0x51)[0xf75a3ea1] /lib32/libc.so.6(abort+0x17e)[0xf75a72ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf758fe37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31607, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7719400] [0xf7719430] /lib32/libc.so.6(gsignal+0x51)[0xf757eea1] /lib32/libc.so.6(abort+0x17e)[0xf75822ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf756ae37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31607, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7706400] [0xf7706430] /lib32/libc.so.6(gsignal+0x51)[0xf756bea1] /lib32/libc.so.6(abort+0x17e)[0xf756f2ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf7557e37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31607, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76ec400] [0xf76ec430] /lib32/libc.so.6(gsignal+0x51)[0xf7551ea1] /lib32/libc.so.6(abort+0x17e)[0xf75552ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf753de37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31607, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Jan 2012 10:34:33	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	699,840	1,442,751	2.0615
16 Jan 2012 11:20:18	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	673,920	1,388,697	2.0606
12 Jan 2012 11:08:54	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	648,000	1,334,504	2.0594
11 Jan 2012 20:04:01	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	622,080	1,279,427	2.0567
09 Jan 2012 15:40:03	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	596,160	1,230,278	2.0637
04 Jan 2012 13:27:17	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	570,240	1,177,636	2.0652
02 Jan 2012 14:21:10	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	544,320	1,123,368	2.0638
29 Dec 2011 19:24:06	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	518,400	1,069,134	2.0624
28 Dec 2011 18:49:25	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	492,480	1,013,896	2.0588
27 Dec 2011 08:22:29	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	466,560	959,441	2.0564
21 Dec 2011 16:49:57	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	440,640	906,533	2.0573
19 Dec 2011 18:00:38	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	414,720	852,358	2.0553
18 Dec 2011 12:48:19	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	388,800	800,035	2.0577
16 Dec 2011 14:33:22	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	362,880	748,106	2.0616
15 Dec 2011 05:43:32	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	336,960	693,144	2.0571
14 Dec 2011 12:13:46	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	311,040	637,712	2.0503
12 Dec 2011 22:01:49	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	285,120	583,567	2.0467
11 Dec 2011 19:44:10	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	259,200	530,500	2.0467
10 Dec 2011 05:10:20	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	233,280	476,795	2.0439
09 Dec 2011 08:52:40	1072022	13626897	hadcm3n_o1dr_1940_40_007543473_1	207,360	423,464	2.0422