Task 15601927

Name	hadcm3n_o1vu_2140_40_008269954_3
Workunit	8425078
Created	9 Feb 2013, 2:19:55 UTC
Sent	9 Feb 2013, 2:25:22 UTC
Report deadline	11 May 2013, 9:52:33 UTC
Received	7 Apr 2013, 22:14:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1237173
Run time	3 days 12 hours 17 min 5 sec
CPU time	3 days 0 hours 34 min 49 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	3.70 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 17:37:46 (38112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:37:54 (38112): No heartbeat from core client for 30 sec - exiting 17:37:55 (38112): No heartbeat from core client for 30 sec - exiting 17:37:56 (38112): No heartbeat from core client for 30 sec - exiting 17:37:57 (38112): No heartbeat from core client for 30 sec - exiting 19:50:22 (38068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:50:25 (38068): No heartbeat from core client for 30 sec - exiting 19:50:26 (38068): No heartbeat from core client for 30 sec - exiting 19:50:27 (38068): No heartbeat from core client for 30 sec - exiting 19:50:28 (38068): No heartbeat from core client for 30 sec - exiting 19:50:29 (38068): No heartbeat from core client for 30 sec - exiting 19:50:30 (38068): No heartbeat from core client for 30 sec - exiting 19:50:31 (38068): No heartbeat from core client for 30 sec - exiting 19:50:32 (38068): No heartbeat from core client for 30 sec - exiting 19:50:33 (38068): No heartbeat from core client for 30 sec - exiting 19:50:34 (38068): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 21:07:04 (34060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:07:05 (34060): No heartbeat from core client for 30 sec - exiting 21:07:06 (34060): No heartbeat from core client for 30 sec - exiting 21:07:07 (34060): No heartbeat from core client for 30 sec - exiting 21:07:08 (34060): No heartbeat from core client for 30 sec - exiting 21:07:09 (34060): No heartbeat from core client for 30 sec - exiting 21:07:10 (34060): No heartbeat from core client for 30 sec - exiting 21:07:11 (34060): No heartbeat from core client for 30 sec - exiting 21:07:12 (34060): No heartbeat from core client for 30 sec - exiting 21:07:13 (34060): No heartbeat from core client for 30 sec - exiting 21:07:14 (34060): No heartbeat from core client for 30 sec - exiting 23:25:30 (38628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:54:41 (38004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:18:22 (41340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:18:16 (38968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:18:21 (38968): No heartbeat from core client for 30 sec - exiting 02:18:22 (38968): No heartbeat from core client for 30 sec - exiting 02:18:23 (38968): No heartbeat from core client for 30 sec - exiting 02:18:24 (38968): No heartbeat from core client for 30 sec - exiting 02:18:25 (38968): No heartbeat from core client for 30 sec - exiting 02:18:26 (38968): No heartbeat from core client for 30 sec - exiting 02:18:27 (38968): No heartbeat from core client for 30 sec - exiting 02:18:28 (38968): No heartbeat from core client for 30 sec - exiting 02:18:29 (38968): No heartbeat from core client for 30 sec - exiting 02:18:30 (38968): No heartbeat from core client for 30 sec - exiting 02:22:11 (25952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:22:15 (25952): No heartbeat from core client for 30 sec - exiting 02:22:16 (25952): No heartbeat from core client for 30 sec - exiting 02:22:17 (25952): No heartbeat from core client for 30 sec - exiting 02:22:18 (25952): No heartbeat from core client for 30 sec - exiting 02:22:19 (25952): No heartbeat from core client for 30 sec - exiting 02:22:20 (25952): No heartbeat from core client for 30 sec - exiting 02:22:21 (25952): No heartbeat from core client for 30 sec - exiting 02:22:22 (25952): No heartbeat from core client for 30 sec - exiting 02:22:23 (25952): No heartbeat from core client for 30 sec - exiting 02:22:24 (25952): No heartbeat from core client for 30 sec - exiting 03:02:11 (42024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:17 (42024): No heartbeat from core client for 30 sec - exiting 03:02:18 (42024): No heartbeat from core client for 30 sec - exiting 03:20:16 (37828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:34 (37828): No heartbeat from core client for 30 sec - exiting 04:01:08 (39512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:01:11 (39512): No heartbeat from core client for 30 sec - exiting 04:06:38 (42200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:53 (43952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:58 (43952): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 07:17:44 (43484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:45 (43484): No heartbeat from core client for 30 sec - exiting 07:17:46 (43484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 07:21:43 (39280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:21:44 (39280): No heartbeat from core client for 30 sec - exiting 07:21:45 (39280): No heartbeat from core client for 30 sec - exiting 07:21:46 (39280): No heartbeat from core client for 30 sec - exiting 07:21:47 (39280): No heartbeat from core client for 30 sec - exiting 07:21:48 (39280): No heartbeat from core client for 30 sec - exiting 07:27:01 (43964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:40:31 (45960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:48:22 (45604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:48:23 (45604): No heartbeat from core client for 30 sec - exiting 10:15:11 (44220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:32:18 (46172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:32:21 (46172): No heartbeat from core client for 30 sec - exiting 10:32:22 (46172): No heartbeat from core client for 30 sec - exiting 10:32:23 (46172): No heartbeat from core client for 30 sec - exiting 10:32:24 (46172): No heartbeat from core client for 30 sec - exiting 10:32:25 (46172): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 10:38:47 (46380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:38:48 (46380): No heartbeat from core client for 30 sec - exiting 10:38:49 (46380): No heartbeat from core client for 30 sec - exiting 10:43:23 (43156): No heartbeat from core client for 30 sec - exiting 10:43:31 (43156): No heartbeat from core client for 30 sec - exiting 10:43:32 (43156): No heartbeat from core client for 30 sec - exiting 10:43:33 (43156): No heartbeat from core client for 30 sec - exiting 10:43:34 (43156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o1vuko.pjy2c10 Error converting file to netcdf: dataout/o1vuko.piy2c10 Error converting file to netcdf: dataout/o1vuko.pfy2c10 Error converting file to netcdf: dataout/o1vuka.phy2c10 Error converting file to netcdf: dataout/o1vuka.pgy2c10 Error converting file to netcdf: dataout/o1vuka.pey2c10 Error converting file to netcdf: dataout/o1vuka.pdy2c10 10:46:05 (47096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:46:06 (47096): No heartbeat from core client for 30 sec - exiting 10:46:07 (47096): No heartbeat from core client for 30 sec - exiting 10:46:08 (47096): No heartbeat from core client for 30 sec - exiting 11:10:27 (44252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:10:37 (44252): No heartbeat from core client for 30 sec - exiting 11:10:38 (44252): No heartbeat from core client for 30 sec - exiting 11:10:39 (44252): No heartbeat from core client for 30 sec - exiting 11:10:40 (44252): No heartbeat from core client for 30 sec - exiting 11:10:41 (44252): No heartbeat from core client for 30 sec - exiting 11:14:54 (46700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:15:01 (46700): No heartbeat from core client for 30 sec - exiting 11:15:02 (46700): No heartbeat from core client for 30 sec - exiting 11:15:03 (46700): No heartbeat from core client for 30 sec - exiting 11:15:04 (46700): No heartbeat from core client for 30 sec - exiting 11:15:05 (46700): No heartbeat from core client for 30 sec - exiting 11:20:33 (46284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:45:25 (47532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:08:27 (46060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:50:19 (46396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:41:50 (50020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:41:57 (50020): No heartbeat from core client for 30 sec - exiting 14:41:58 (50020): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 15:53:44 (36320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:53:45 (36320): No heartbeat from core client for 30 sec - exiting 15:53:46 (36320): No heartbeat from core client for 30 sec - exiting 15:53:47 (36320): No heartbeat from core client for 30 sec - exiting 15:53:48 (36320): No heartbeat from core client for 30 sec - exiting 15:53:49 (36320): No heartbeat from core client for 30 sec - exiting 15:58:34 (48684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:58:37 (48684): No heartbeat from core client for 30 sec - exiting 15:58:38 (48684): No heartbeat from core client for 30 sec - exiting 15:58:39 (48684): No heartbeat from core client for 30 sec - exiting 15:58:40 (48684): No heartbeat from core client for 30 sec - exiting 15:58:41 (48684): No heartbeat from core client for 30 sec - exiting 15:58:42 (48684): No heartbeat from core client for 30 sec - exiting 15:58:43 (48684): No heartbeat from core client for 30 sec - exiting 15:58:44 (48684): No heartbeat from core client for 30 sec - exiting 15:58:45 (48684): No heartbeat from core client for 30 sec - exiting 15:58:46 (48684): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 18:22:49 (46200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:22:50 (46200): No heartbeat from core client for 30 sec - exiting 18:22:51 (46200): No heartbeat from core client for 30 sec - exiting 18:22:52 (46200): No heartbeat from core client for 30 sec - exiting 18:22:53 (46200): No heartbeat from core client for 30 sec - exiting 18:22:54 (46200): No heartbeat from core client for 30 sec - exiting 18:22:55 (46200): No heartbeat from core client for 30 sec - exiting 18:22:56 (46200): No heartbeat from core client for 30 sec - exiting 18:22:57 (46200): No heartbeat from core client for 30 sec - exiting 18:28:03 (44792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:28:05 (44792): No heartbeat from core client for 30 sec - exiting 18:28:06 (44792): No heartbeat from core client for 30 sec - exiting 18:28:07 (44792): No heartbeat from core client for 30 sec - exiting 18:28:08 (44792): No heartbeat from core client for 30 sec - exiting 18:28:09 (44792): No heartbeat from core client for 30 sec - exiting 18:28:10 (44792): No heartbeat from core client for 30 sec - exiting 18:28:11 (44792): No heartbeat from core client for 30 sec - exiting 18:28:12 (44792): No heartbeat from core client for 30 sec - exiting 18:28:13 (44792): No heartbeat from core client for 30 sec - exiting 18:28:14 (44792): No heartbeat from core client for 30 sec - exiting 18:31:56 (51968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:57 (51968): No heartbeat from core client for 30 sec - exiting 18:31:58 (51968): No heartbeat from core client for 30 sec - exiting 18:35:48 (48480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:48 (40580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:58 (40580): No heartbeat from core client for 30 sec - exiting 18:38:59 (40580): No heartbeat from core client for 30 sec - exiting 18:39:00 (40580): No heartbeat from core client for 30 sec - exiting 18:39:01 (40580): No heartbeat from core client for 30 sec - exiting 18:39:02 (40580): No heartbeat from core client for 30 sec - exiting 18:39:03 (40580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:28:59 (52300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:29:03 (52300): No heartbeat from core client for 30 sec - exiting 00:29:04 (52300): No heartbeat from core client for 30 sec - exiting 00:29:05 (52300): No heartbeat from core client for 30 sec - exiting 00:29:06 (52300): No heartbeat from core client for 30 sec - exiting 00:29:07 (52300): No heartbeat from core client for 30 sec - exiting 00:29:08 (52300): No heartbeat from core client for 30 sec - exiting 00:29:09 (52300): No heartbeat from core client for 30 sec - exiting 02:47:43 (53464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:47:55 (53464): No heartbeat from core client for 30 sec - exiting 02:47:56 (53464): No heartbeat from core client for 30 sec - exiting 02:47:57 (53464): No heartbeat from core client for 30 sec - exiting 02:47:58 (53464): No heartbeat from core client for 30 sec - exiting 02:47:59 (53464): No heartbeat from core client for 30 sec - exiting 02:48:00 (53464): No heartbeat from core client for 30 sec - exiting 02:48:01 (53464): No heartbeat from core client for 30 sec - exiting 02:48:02 (53464): No heartbeat from core client for 30 sec - exiting 02:48:03 (53464): No heartbeat from core client for 30 sec - exiting 02:48:04 (53464): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 04:02:00 (54300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:02:01 (54300): No heartbeat from core client for 30 sec - exiting 04:02:02 (54300): No heartbeat from core client for 30 sec - exiting 04:02:03 (54300): No heartbeat from core client for 30 sec - exiting 05:08:30 (52332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:18:21 (56028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:18:24 (56028): No heartbeat from core client for 30 sec - exiting 05:18:25 (56028): No heartbeat from core client for 30 sec - exiting 05:18:26 (56028): No heartbeat from core client for 30 sec - exiting 05:18:27 (56028): No heartbeat from core client for 30 sec - exiting 05:18:28 (56028): No heartbeat from core client for 30 sec - exiting 05:18:29 (56028): No heartbeat from core client for 30 sec - exiting 06:22:20 (53764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 07:21:11 (55664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:21:12 (55664): No heartbeat from core client for 30 sec - exiting 07:21:13 (55664): No heartbeat from core client for 30 sec - exiting 07:21:14 (55664): No heartbeat from core client for 30 sec - exiting 07:21:15 (55664): No heartbeat from core client for 30 sec - exiting 07:21:16 (55664): No heartbeat from core client for 30 sec - exiting 07:21:17 (55664): No heartbeat from core client for 30 sec - exiting 09:50:28 (55172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:17 (55512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:23 (55512): No heartbeat from core client for 30 sec - exiting 10:06:24 (55512): No heartbeat from core client for 30 sec - exiting 12:00:27 (55912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 14:45:49 (58140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:45:50 (58140): No heartbeat from core client for 30 sec - exiting 14:45:51 (58140): No heartbeat from core client for 30 sec - exiting 14:45:52 (58140): No heartbeat from core client for 30 sec - exiting 14:45:53 (58140): No heartbeat from core client for 30 sec - exiting 14:45:54 (58140): No heartbeat from core client for 30 sec - exiting 14:45:55 (58140): No heartbeat from core client for 30 sec - exiting 14:45:56 (58140): No heartbeat from core client for 30 sec - exiting 14:45:57 (58140): No heartbeat from core client for 30 sec - exiting 14:45:58 (58140): No heartbeat from core client for 30 sec - exiting 14:45:59 (58140): No heartbeat from core client for 30 sec - exiting 15:21:36 (56508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:21:37 (56508): No heartbeat from core client for 30 sec - exiting 15:21:38 (56508): No heartbeat from core client for 30 sec - exiting 15:21:39 (56508): No heartbeat from core client for 30 sec - exiting 15:21:40 (56508): No heartbeat from core client for 30 sec - exiting 15:21:41 (56508): No heartbeat from core client for 30 sec - exiting 15:21:42 (56508): No heartbeat from core client for 30 sec - exiting 16:05:11 (58204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:47:42 (58376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:41 (56788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:06:10 (60084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:06:11 (60084): No heartbeat from core client for 30 sec - exiting 18:06:12 (60084): No heartbeat from core client for 30 sec - exiting 18:06:13 (60084): No heartbeat from core client for 30 sec - exiting 18:06:14 (60084): No heartbeat from core client for 30 sec - exiting 18:06:15 (60084): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 18:13:11 (58780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:12 (58780): No heartbeat from core client for 30 sec - exiting 18:13:13 (58780): No heartbeat from core client for 30 sec - exiting 18:13:14 (58780): No heartbeat from core client for 30 sec - exiting 18:13:15 (58780): No heartbeat from core client for 30 sec - exiting 18:13:16 (58780): No heartbeat from core client for 30 sec - exiting 18:13:17 (58780): No heartbeat from core client for 30 sec - exiting 18:13:18 (58780): No heartbeat from core client for 30 sec - exiting 18:13:19 (58780): No heartbeat from core client for 30 sec - exiting 18:13:20 (58780): No heartbeat from core client for 30 sec - exiting 18:13:21 (58780): No heartbeat from core client for 30 sec - exiting 18:21:26 (60352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:21:27 (60352): No heartbeat from core client for 30 sec - exiting 18:21:28 (60352): No heartbeat from core client for 30 sec - exiting 18:21:29 (60352): No heartbeat from core client for 30 sec - exiting 18:21:30 (60352): No heartbeat from core client for 30 sec - exiting 18:21:31 (60352): No heartbeat from core client for 30 sec - exiting 18:21:32 (60352): No heartbeat from core client for 30 sec - exiting 18:30:18 (59592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:22 (59592): No heartbeat from core client for 30 sec - exiting 18:30:23 (59592): No heartbeat from core client for 30 sec - exiting 18:30:24 (59592): No heartbeat from core client for 30 sec - exiting 18:30:25 (59592): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 19:02:49 (59828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:02:50 (59828): No heartbeat from core client for 30 sec - exiting 19:10:11 (61068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:10:12 (61068): No heartbeat from core client for 30 sec - exiting 19:10:13 (61068): No heartbeat from core client for 30 sec - exiting 19:10:14 (61068): No heartbeat from core client for 30 sec - exiting 19:19:54 (59380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:32:03 (22160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:48:24 (7172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:09:10 (1208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:34:21 (4324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:37:19 (9252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:57:32 (10312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:40 (588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:47 (588): No heartbeat from core client for 30 sec - exiting 17:58:48 (588): No heartbeat from core client for 30 sec - exiting 17:58:49 (588): No heartbeat from core client for 30 sec - exiting 17:58:50 (588): No heartbeat from core client for 30 sec - exiting 17:58:51 (588): No heartbeat from core client for 30 sec - exiting 17:58:52 (588): No heartbeat from core client for 30 sec - exiting 17:58:53 (588): No heartbeat from core client for 30 sec - exiting 17:58:54 (588): No heartbeat from core client for 30 sec - exiting 17:58:55 (588): No heartbeat from core client for 30 sec - exiting 17:58:56 (588): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 18:04:41 (1584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:04:42 (1584): No heartbeat from core client for 30 sec - exiting 18:10:42 (16432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17328, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17328, iMonCtr=1 Model crash detected, will try to restart... 02:48:45 (17328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:53:54 (25380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:57:00 (22520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:08:05 (21980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:15:22 (24120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:26:42 (26140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:39:46 (27236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:52 (27272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:53:05 (26040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:06:09 (25216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:21:54 (27360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:28:25 (15568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:38:03 (24676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:26:44 (24832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24972, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18568, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18568, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38920, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Apr 2013 21:20:00	1237173	15601927	hadcm3n_o1vu_2140_40_008269954_3	285,120	258,526	0.9067
04 Apr 2013 10:21:15	1237173	15601927	hadcm3n_o1vu_2140_40_008269954_3	259,200	231,931	0.8948
04 Apr 2013 00:42:34	1237173	15601927	hadcm3n_o1vu_2140_40_008269954_3	233,280	206,634	0.8858
03 Apr 2013 18:52:09	1237173	15601927	hadcm3n_o1vu_2140_40_008269954_3	207,360	185,983	0.8969
03 Apr 2013 12:57:13	1237173	15601927	hadcm3n_o1vu_2140_40_008269954_3	181,440	165,211	0.9106
31 Mar 2013 22:40:42	1237173	15601927	hadcm3n_o1vu_2140_40_008269954_3	155,520	143,780	0.9245
31 Mar 2013 16:46:12	1237173	15601927	hadcm3n_o1vu_2140_40_008269954_3	129,600	121,673	0.9388
31 Mar 2013 09:19:17	1237173	15601927	hadcm3n_o1vu_2140_40_008269954_3	103,680	99,105	0.9559
30 Mar 2013 23:01:25	1237173	15601927	hadcm3n_o1vu_2140_40_008269954_3	77,760	76,464	0.9833
30 Mar 2013 15:45:40	1237173	15601927	hadcm3n_o1vu_2140_40_008269954_3	51,840	53,710	1.0361
30 Mar 2013 06:23:39	1237173	15601927	hadcm3n_o1vu_2140_40_008269954_3	25,920	27,425	1.0581