Task 15671339

Name	hadcm3n_zmnq_1920_40_008316927_3
Workunit	8468062
Created	18 Mar 2013, 13:32:54 UTC
Sent	18 Mar 2013, 13:33:02 UTC
Report deadline	17 Jun 2013, 21:00:13 UTC
Received	26 Apr 2013, 11:21:55 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1217831
Run time	24 days 7 hours 14 min 25 sec
CPU time	15 days 10 hours 52 min 30 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:21:44 (388500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:14:57 (5156): No heartbeat from core client for 30 sec - exiting 13:14:58 (5156): No heartbeat from core client for 30 sec - exiting 13:14:59 (5156): No heartbeat from core client for 30 sec - exiting 13:15:00 (5156): No heartbeat from core client for 30 sec - exiting 13:15:01 (5156): No heartbeat from core client for 30 sec - exiting 13:15:02 (5156): No heartbeat from core client for 30 sec - exiting 13:15:03 (5156): No heartbeat from core client for 30 sec - exiting 13:15:04 (5156): No heartbeat from core client for 30 sec - exiting 13:15:05 (5156): No heartbeat from core client for 30 sec - exiting 13:15:06 (5156): No heartbeat from core client for 30 sec - exiting 13:15:07 (5156): No heartbeat from core client for 30 sec - exiting 13:15:08 (5156): No heartbeat from core client for 30 sec - exiting 13:15:09 (5156): No heartbeat from core client for 30 sec - exiting 13:15:10 (5156): No heartbeat from core client for 30 sec - exiting 13:15:11 (5156): No heartbeat from core client for 30 sec - exiting 13:15:12 (5156): No heartbeat from core client for 30 sec - exiting 13:15:13 (5156): No heartbeat from core client for 30 sec - exiting 13:15:14 (5156): No heartbeat from core client for 30 sec - exiting 13:15:15 (5156): No heartbeat from core client for 30 sec - exiting 13:15:16 (5156): No heartbeat from core client for 30 sec - exiting 13:15:17 (5156): No heartbeat from core client for 30 sec - exiting 13:15:18 (5156): No heartbeat from core client for 30 sec - exiting 13:15:19 (5156): No heartbeat from core client for 30 sec - exiting 13:15:20 (5156): No heartbeat from core client for 30 sec - exiting 13:15:21 (5156): No heartbeat from core client for 30 sec - exiting 13:15:22 (5156): No heartbeat from core client for 30 sec - exiting 13:15:23 (5156): No heartbeat from core client for 30 sec - exiting 13:15:24 (5156): No heartbeat from core client for 30 sec - exiting 13:15:25 (5156): No heartbeat from core client for 30 sec - exiting 13:15:26 (5156): No heartbeat from core client for 30 sec - exiting 13:15:27 (5156): No heartbeat from core client for 30 sec - exiting 13:15:28 (5156): No heartbeat from core client for 30 sec - exiting 13:15:29 (5156): No heartbeat from core client for 30 sec - exiting 13:15:30 (5156): No heartbeat from core client for 30 sec - exiting 13:15:31 (5156): No heartbeat from core client for 30 sec - exiting 13:15:32 (5156): No heartbeat from core client for 30 sec - exiting 13:15:33 (5156): No heartbeat from core client for 30 sec - exiting 13:15:34 (5156): No heartbeat from core client for 30 sec - exiting 13:15:35 (5156): No heartbeat from core client for 30 sec - exiting 13:15:36 (5156): No heartbeat from core client for 30 sec - exiting 13:15:37 (5156): No heartbeat from core client for 30 sec - exiting 13:15:38 (5156): No heartbeat from core client for 30 sec - exiting 13:15:39 (5156): No heartbeat from core client for 30 sec - exiting 13:15:40 (5156): No heartbeat from core client for 30 sec - exiting 13:15:41 (5156): No heartbeat from core client for 30 sec - exiting 13:15:42 (5156): No heartbeat from core client for 30 sec - exiting 13:15:43 (5156): No heartbeat from core client for 30 sec - exiting 13:15:44 (5156): No heartbeat from core client for 30 sec - exiting 13:15:45 (5156): No heartbeat from core client for 30 sec - exiting 13:15:46 (5156): No heartbeat from core client for 30 sec - exiting 13:15:47 (5156): No heartbeat from core client for 30 sec - exiting 13:15:48 (5156): No heartbeat from core client for 30 sec - exiting 13:15:49 (5156): No heartbeat from core client for 30 sec - exiting 13:15:50 (5156): No heartbeat from core client for 30 sec - exiting 13:15:51 (5156): No heartbeat from core client for 30 sec - exiting 13:15:52 (5156): No heartbeat from core client for 30 sec - exiting 13:15:53 (5156): No heartbeat from core client for 30 sec - exiting 13:15:54 (5156): No heartbeat from core client for 30 sec - exiting 13:15:55 (5156): No heartbeat from core client for 30 sec - exiting 13:15:56 (5156): No heartbeat from core client for 30 sec - exiting 13:15:57 (5156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:35:46 (86240): No heartbeat from core client for 30 sec - exiting 12:35:47 (86240): No heartbeat from core client for 30 sec - exiting 12:35:48 (86240): No heartbeat from core client for 30 sec - exiting 12:35:49 (86240): No heartbeat from core client for 30 sec - exiting 12:35:50 (86240): No heartbeat from core client for 30 sec - exiting 12:35:51 (86240): No heartbeat from core client for 30 sec - exiting 12:35:52 (86240): No heartbeat from core client for 30 sec - exiting 12:35:53 (86240): No heartbeat from core client for 30 sec - exiting 12:35:54 (86240): No heartbeat from core client for 30 sec - exiting 12:35:55 (86240): No heartbeat from core client for 30 sec - exiting 12:35:56 (86240): No heartbeat from core client for 30 sec - exiting 12:35:57 (86240): No heartbeat from core client for 30 sec - exiting 12:35:58 (86240): No heartbeat from core client for 30 sec - exiting 12:35:59 (86240): No heartbeat from core client for 30 sec - exiting 12:36:00 (86240): No heartbeat from core client for 30 sec - exiting 12:36:01 (86240): No heartbeat from core client for 30 sec - exiting 12:36:02 (86240): No heartbeat from core client for 30 sec - exiting 12:36:03 (86240): No heartbeat from core client for 30 sec - exiting 12:36:04 (86240): No heartbeat from core client for 30 sec - exiting 12:36:05 (86240): No heartbeat from core client for 30 sec - exiting 12:36:06 (86240): No heartbeat from core client for 30 sec - exiting 12:36:07 (86240): No heartbeat from core client for 30 sec - exiting 12:36:08 (86240): No heartbeat from core client for 30 sec - exiting 12:36:09 (86240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:15:15 (48460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=125792, iMonCtr=1 Model crash detected, will try to restart... 13:05:56 (4352): No heartbeat from core client for 30 sec - exiting 13:05:57 (4352): No heartbeat from core client for 30 sec - exiting 13:05:58 (4352): No heartbeat from core client for 30 sec - exiting 13:05:59 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:09:48 (664): No heartbeat from core client for 30 sec - exiting 14:09:49 (664): No heartbeat from core client for 30 sec - exiting 14:09:50 (664): No heartbeat from core client for 30 sec - exiting 14:09:51 (664): No heartbeat from core client for 30 sec - exiting 14:09:52 (664): No heartbeat from core client for 30 sec - exiting 14:09:53 (664): No heartbeat from core client for 30 sec - exiting 14:09:54 (664): No heartbeat from core client for 30 sec - exiting 14:09:55 (664): No heartbeat from core client for 30 sec - exiting 14:09:56 (664): No heartbeat from core client for 30 sec - exiting 14:09:57 (664): No heartbeat from core client for 30 sec - exiting 14:09:58 (664): No heartbeat from core client for 30 sec - exiting 14:09:59 (664): No heartbeat from core client for 30 sec - exiting 14:10:00 (664): No heartbeat from core client for 30 sec - exiting 14:10:01 (664): No heartbeat from core client for 30 sec - exiting 14:10:02 (664): No heartbeat from core client for 30 sec - exiting 14:10:03 (664): No heartbeat from core client for 30 sec - exiting 14:10:04 (664): No heartbeat from core client for 30 sec - exiting 14:10:05 (664): No heartbeat from core client for 30 sec - exiting 14:10:06 (664): No heartbeat from core client for 30 sec - exiting 14:10:07 (664): No heartbeat from core client for 30 sec - exiting 14:10:08 (664): No heartbeat from core client for 30 sec - exiting 14:10:09 (664): No heartbeat from core client for 30 sec - exiting 14:10:10 (664): No heartbeat from core client for 30 sec - exiting 14:10:11 (664): No heartbeat from core client for 30 sec - exiting 14:10:12 (664): No heartbeat from core client for 30 sec - exiting 14:10:13 (664): No heartbeat from core client for 30 sec - exiting 14:10:14 (664): No heartbeat from core client for 30 sec - exiting 14:10:15 (664): No heartbeat from core client for 30 sec - exiting 14:10:16 (664): No heartbeat from core client for 30 sec - exiting 14:10:17 (664): No heartbeat from core client for 30 sec - exiting 14:10:18 (664): No heartbeat from core client for 30 sec - exiting 14:10:19 (664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:10:29 (34804): No heartbeat from core client for 30 sec - exiting 13:10:30 (34804): No heartbeat from core client for 30 sec - exiting 13:10:31 (34804): No heartbeat from core client for 30 sec - exiting 13:10:32 (34804): No heartbeat from core client for 30 sec - exiting 13:10:33 (34804): No heartbeat from core client for 30 sec - exiting 13:10:34 (34804): No heartbeat from core client for 30 sec - exiting 13:10:35 (34804): No heartbeat from core client for 30 sec - exiting 13:10:36 (34804): No heartbeat from core client for 30 sec - exiting 13:10:38 (34804): No heartbeat from core client for 30 sec - exiting 13:10:39 (34804): No heartbeat from core client for 30 sec - exiting 13:10:40 (34804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:10:41 (34804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:28:05 (4100): No heartbeat from core client for 30 sec - exiting 19:28:06 (4100): No heartbeat from core client for 30 sec - exiting 19:28:07 (4100): No heartbeat from core client for 30 sec - exiting 19:28:08 (4100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:58:37 (148752): No heartbeat from core client for 30 sec - exiting 12:58:38 (148752): No heartbeat from core client for 30 sec - exiting 12:58:39 (148752): No heartbeat from core client for 30 sec - exiting 12:58:40 (148752): No heartbeat from core client for 30 sec - exiting 12:58:41 (148752): No heartbeat from core client for 30 sec - exiting 12:58:42 (148752): No heartbeat from core client for 30 sec - exiting 12:58:43 (148752): No heartbeat from core client for 30 sec - exiting 12:58:44 (148752): No heartbeat from core client for 30 sec - exiting 12:58:45 (148752): No heartbeat from core client for 30 sec - exiting 12:58:46 (148752): No heartbeat from core client for 30 sec - exiting 12:58:47 (148752): No heartbeat from core client for 30 sec - exiting 12:58:48 (148752): No heartbeat from core client for 30 sec - exiting 12:58:49 (148752): No heartbeat from core client for 30 sec - exiting 12:58:50 (148752): No heartbeat from core client for 30 sec - exiting 12:58:51 (148752): No heartbeat from core client for 30 sec - exiting 12:58:52 (148752): No heartbeat from core client for 30 sec - exiting 12:58:53 (148752): No heartbeat from core client for 30 sec - exiting 12:58:54 (148752): No heartbeat from core client for 30 sec - exiting 12:58:55 (148752): No heartbeat from core client for 30 sec - exiting 12:58:56 (148752): No heartbeat from core client for 30 sec - exiting 12:58:57 (148752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:58:58 (148752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:27:43 (5804): No heartbeat from core client for 30 sec - exiting 20:27:44 (5804): No heartbeat from core client for 30 sec - exiting 20:27:45 (5804): No heartbeat from core client for 30 sec - exiting 20:27:46 (5804): No heartbeat from core client for 30 sec - exiting 20:27:47 (5804): No heartbeat from core client for 30 sec - exiting 20:27:48 (5804): No heartbeat from core client for 30 sec - exiting 20:27:49 (5804): No heartbeat from core client for 30 sec - exiting 20:27:50 (5804): No heartbeat from core client for 30 sec - exiting 20:27:51 (5804): No heartbeat from core client for 30 sec - exiting 20:27:52 (5804): No heartbeat from core client for 30 sec - exiting 20:27:53 (5804): No heartbeat from core client for 30 sec - exiting 20:27:54 (5804): No heartbeat from core client for 30 sec - exiting 20:27:55 (5804): No heartbeat from core client for 30 sec - exiting 20:27:56 (5804): No heartbeat from core client for 30 sec - exiting 20:27:57 (5804): No heartbeat from core client for 30 sec - exiting 20:27:58 (5804): No heartbeat from core client for 30 sec - exiting 20:27:59 (5804): No heartbeat from core client for 30 sec - exiting 20:28:00 (5804): No heartbeat from core client for 30 sec - exiting 20:28:01 (5804): No heartbeat from core client for 30 sec - exiting 20:28:02 (5804): No heartbeat from core client for 30 sec - exiting 20:28:03 (5804): No heartbeat from core client for 30 sec - exiting 20:28:04 (5804): No heartbeat from core client for 30 sec - exiting 20:28:05 (5804): No heartbeat from core client for 30 sec - exiting 20:28:06 (5804): No heartbeat from core client for 30 sec - exiting 20:28:07 (5804): No heartbeat from core client for 30 sec - exiting 20:28:08 (5804): No heartbeat from core client for 30 sec - exiting 20:28:09 (5804): No heartbeat from core client for 30 sec - exiting 20:28:10 (5804): No heartbeat from core client for 30 sec - exiting 20:28:11 (5804): No heartbeat from core client for 30 sec - exiting 20:28:12 (5804): No heartbeat from core client for 30 sec - exiting 20:28:13 (5804): No heartbeat from core client for 30 sec - exiting 20:28:14 (5804): No heartbeat from core client for 30 sec - exiting 20:28:15 (5804): No heartbeat from core client for 30 sec - exiting 20:28:16 (5804): No heartbeat from core client for 30 sec - exiting 20:28:17 (5804): No heartbeat from core client for 30 sec - exiting 20:28:18 (5804): No heartbeat from core client for 30 sec - exiting 20:28:19 (5804): No heartbeat from core client for 30 sec - exiting 20:28:20 (5804): No heartbeat from core client for 30 sec - exiting 20:28:21 (5804): No heartbeat from core client for 30 sec - exiting 20:28:22 (5804): No heartbeat from core client for 30 sec - exiting 20:28:23 (5804): No heartbeat from core client for 30 sec - exiting 20:28:24 (5804): No heartbeat from core client for 30 sec - exiting 20:28:25 (5804): No heartbeat from core client for 30 sec - exiting 20:28:26 (5804): No heartbeat from core client for 30 sec - exiting 20:28:27 (5804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:28 (5804): No heartbeat from core client for 30 sec - exiting 20:28:29 (5804): No heartbeat from core client for 30 sec - exiting 20:28:30 (5804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zmnqko.pjd5c10 Error converting file to netcdf: dataout/zmnqko.pid5c10 Error converting file to netcdf: dataout/zmnqko.pfd5c10 Error converting file to netcdf: dataout/zmnqka.phd5c10 Error converting file to netcdf: dataout/zmnqka.pgd5c10 Error converting file to netcdf: dataout/zmnqka.ped5c10 Error converting file to netcdf: dataout/zmnqka.pdd5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:44:08 (26960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=41420, iMonCtr=1 Model crash detected, will try to restart... 10:16:17 (14452): No heartbeat from core client for 30 sec - exiting 10:16:19 (14452): No heartbeat from core client for 30 sec - exiting 10:16:20 (14452): No heartbeat from core client for 30 sec - exiting 10:16:21 (14452): No heartbeat from core client for 30 sec - exiting 10:16:22 (14452): No heartbeat from core client for 30 sec - exiting 10:16:23 (14452): No heartbeat from core client for 30 sec - exiting 10:16:24 (14452): No heartbeat from core client for 30 sec - exiting 10:16:25 (14452): No heartbeat from core client for 30 sec - exiting 10:16:26 (14452): No heartbeat from core client for 30 sec - exiting 10:16:27 (14452): No heartbeat from core client for 30 sec - exiting 10:16:28 (14452): No heartbeat from core client for 30 sec - exiting 10:16:29 (14452): No heartbeat from core client for 30 sec - exiting 10:16:30 (14452): No heartbeat from core client for 30 sec - exiting 10:16:31 (14452): No heartbeat from core client for 30 sec - exiting 10:16:32 (14452): No heartbeat from core client for 30 sec - exiting 10:16:33 (14452): No heartbeat from core client for 30 sec - exiting 10:16:34 (14452): No heartbeat from core client for 30 sec - exiting 10:16:35 (14452): No heartbeat from core client for 30 sec - exiting 10:16:36 (14452): No heartbeat from core client for 30 sec - exiting 10:16:37 (14452): No heartbeat from core client for 30 sec - exiting 10:16:38 (14452): No heartbeat from core client for 30 sec - exiting 10:16:39 (14452): No heartbeat from core client for 30 sec - exiting 10:16:40 (14452): No heartbeat from core client for 30 sec - exiting 10:16:41 (14452): No heartbeat from core client for 30 sec - exiting 10:16:42 (14452): No heartbeat from core client for 30 sec - exiting 10:16:43 (14452): No heartbeat from core client for 30 sec - exiting 10:16:44 (14452): No heartbeat from core client for 30 sec - exiting 10:16:45 (14452): No heartbeat from core client for 30 sec - exiting 10:16:46 (14452): No heartbeat from core client for 30 sec - exiting 10:16:47 (14452): No heartbeat from core client for 30 sec - exiting 10:16:48 (14452): No heartbeat from core client for 30 sec - exiting 10:16:49 (14452): No heartbeat from core client for 30 sec - exiting 10:16:50 (14452): No heartbeat from core client for 30 sec - exiting 10:16:51 (14452): No heartbeat from core client for 30 sec - exiting 10:16:52 (14452): No heartbeat from core client for 30 sec - exiting 10:16:53 (14452): No heartbeat from core client for 30 sec - exiting 10:16:54 (14452): No heartbeat from core client for 30 sec - exiting 10:16:55 (14452): No heartbeat from core client for 30 sec - exiting 10:16:56 (14452): No heartbeat from core client for 30 sec - exiting 10:16:57 (14452): No heartbeat from core client for 30 sec - exiting 10:16:58 (14452): No heartbeat from core client for 30 sec - exiting 10:16:59 (14452): No heartbeat from core client for 30 sec - exiting 10:17:00 (14452): No heartbeat from core client for 30 sec - exiting 10:17:01 (14452): No heartbeat from core client for 30 sec - exiting 10:17:02 (14452): No heartbeat from core client for 30 sec - exiting 10:17:03 (14452): No heartbeat from core client for 30 sec - exiting 10:17:04 (14452): No heartbeat from core client for 30 sec - exiting 10:17:05 (14452): No heartbeat from core client for 30 sec - exiting 10:17:06 (14452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:29:24 (85072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:46:28 (115280): No heartbeat from core client for 30 sec - exiting 15:46:30 (115280): No heartbeat from core client for 30 sec - exiting 15:46:31 (115280): No heartbeat from core client for 30 sec - exiting 15:46:32 (115280): No heartbeat from core client for 30 sec - exiting 15:46:33 (115280): No heartbeat from core client for 30 sec - exiting 15:46:34 (115280): No heartbeat from core client for 30 sec - exiting 15:46:35 (115280): No heartbeat from core client for 30 sec - exiting 15:46:36 (115280): No heartbeat from core client for 30 sec - exiting 15:46:37 (115280): No heartbeat from core client for 30 sec - exiting 15:46:38 (115280): No heartbeat from core client for 30 sec - exiting 15:46:39 (115280): No heartbeat from core client for 30 sec - exiting 15:46:40 (115280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:55:12 (7752): No heartbeat from core client for 30 sec - exiting 13:55:13 (7752): No heartbeat from core client for 30 sec - exiting 13:55:14 (7752): No heartbeat from core client for 30 sec - exiting 13:55:15 (7752): No heartbeat from core client for 30 sec - exiting 13:55:16 (7752): No heartbeat from core client for 30 sec - exiting 13:55:17 (7752): No heartbeat from core client for 30 sec - exiting 13:55:18 (7752): No heartbeat from core client for 30 sec - exiting 13:55:19 (7752): No heartbeat from core client for 30 sec - exiting 13:55:20 (7752): No heartbeat from core client for 30 sec - exiting 13:55:21 (7752): No heartbeat from core client for 30 sec - exiting 13:55:22 (7752): No heartbeat from core client for 30 sec - exiting 13:55:23 (7752): No heartbeat from core client for 30 sec - exiting 13:55:24 (7752): No heartbeat from core client for 30 sec - exiting 13:55:25 (7752): No heartbeat from core client for 30 sec - exiting 13:55:26 (7752): No heartbeat from core client for 30 sec - exiting 13:55:27 (7752): No heartbeat from core client for 30 sec - exiting 13:55:28 (7752): No heartbeat from core client for 30 sec - exiting 13:55:29 (7752): No heartbeat from core client for 30 sec - exiting 13:55:30 (7752): No heartbeat from core client for 30 sec - exiting 13:55:31 (7752): No heartbeat from core client for 30 sec - exiting 13:55:32 (7752): No heartbeat from core client for 30 sec - exiting 13:55:33 (7752): No heartbeat from core client for 30 sec - exiting 13:55:34 (7752): No heartbeat from core client for 30 sec - exiting 13:55:35 (7752): No heartbeat from core client for 30 sec - exiting 13:55:36 (7752): No heartbeat from core client for 30 sec - exiting 13:55:37 (7752): No heartbeat from core client for 30 sec - exiting 13:55:38 (7752): No heartbeat from core client for 30 sec - exiting 13:55:39 (7752): No heartbeat from core client for 30 sec - exiting 13:55:40 (7752): No heartbeat from core client for 30 sec - exiting 13:55:41 (7752): No heartbeat from core client for 30 sec - exiting 13:55:42 (7752): No heartbeat from core client for 30 sec - exiting 13:55:43 (7752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:42:48 (4932): No heartbeat from core client for 30 sec - exiting 15:42:49 (4932): No heartbeat from core client for 30 sec - exiting 15:42:50 (4932): No heartbeat from core client for 30 sec - exiting 15:42:51 (4932): No heartbeat from core client for 30 sec - exiting 15:42:52 (4932): No heartbeat from core client for 30 sec - exiting 15:42:53 (4932): No heartbeat from core client for 30 sec - exiting 15:42:54 (4932): No heartbeat from core client for 30 sec - exiting 15:42:55 (4932): No heartbeat from core client for 30 sec - exiting 15:42:56 (4932): No heartbeat from core client for 30 sec - exiting 15:42:57 (4932): No heartbeat from core client for 30 sec - exiting 15:42:58 (4932): No heartbeat from core client for 30 sec - exiting 15:42:59 (4932): No heartbeat from core client for 30 sec - exiting 15:43:00 (4932): No heartbeat from core client for 30 sec - exiting 15:43:01 (4932): No heartbeat from core client for 30 sec - exiting 15:43:02 (4932): No heartbeat from core client for 30 sec - exiting 15:43:03 (4932): No heartbeat from core client for 30 sec - exiting 15:43:04 (4932): No heartbeat from core client for 30 sec - exiting 15:43:05 (4932): No heartbeat from core client for 30 sec - exiting 15:43:06 (4932): No heartbeat from core client for 30 sec - exiting 15:43:07 (4932): No heartbeat from core client for 30 sec - exiting 15:43:08 (4932): No heartbeat from core client for 30 sec - exiting 15:43:09 (4932): No heartbeat from core client for 30 sec - exiting 15:43:10 (4932): No heartbeat from core client for 30 sec - exiting 15:43:11 (4932): No heartbeat from core client for 30 sec - exiting 15:43:12 (4932): No heartbeat from core client for 30 sec - exiting 15:43:13 (4932): No heartbeat from core client for 30 sec - exiting 15:43:14 (4932): No heartbeat from core client for 30 sec - exiting 15:43:15 (4932): No heartbeat from core client for 30 sec - exiting 15:43:16 (4932): No heartbeat from core client for 30 sec - exiting 15:43:17 (4932): No heartbeat from core client for 30 sec - exiting 15:43:18 (4932): No heartbeat from core client for 30 sec - exiting 15:43:19 (4932): No heartbeat from core client for 30 sec - exiting 15:43:20 (4932): No heartbeat from core client for 30 sec - exiting 15:43:21 (4932): No heartbeat from core client for 30 sec - exiting 15:43:22 (4932): No heartbeat from core client for 30 sec - exiting 15:43:23 (4932): No heartbeat from core client for 30 sec - exiting 15:43:24 (4932): No heartbeat from core client for 30 sec - exiting 15:43:25 (4932): No heartbeat from core client for 30 sec - exiting 15:43:26 (4932): No heartbeat from core client for 30 sec - exiting 15:43:27 (4932): No heartbeat from core client for 30 sec - exiting 15:43:28 (4932): No heartbeat from core client for 30 sec - exiting 15:43:29 (4932): No heartbeat from core client for 30 sec - exiting 15:43:30 (4932): No heartbeat from core client for 30 sec - exiting 15:43:31 (4932): No heartbeat from core client for 30 sec - exiting 15:43:32 (4932): No heartbeat from core client for 30 sec - exiting 15:43:33 (4932): No heartbeat from core client for 30 sec - exiting 15:43:34 (4932): No heartbeat from core client for 30 sec - exiting 15:43:35 (4932): No heartbeat from core client for 30 sec - exiting 15:43:36 (4932): No heartbeat from core client for 30 sec - exiting 15:43:37 (4932): No heartbeat from core client for 30 sec - exiting 15:43:38 (4932): No heartbeat from core client for 30 sec - exiting 15:43:39 (4932): No heartbeat from core client for 30 sec - exiting 15:43:40 (4932): No heartbeat from core client for 30 sec - exiting 15:43:41 (4932): No heartbeat from core client for 30 sec - exiting 15:43:42 (4932): No heartbeat from core client for 30 sec - exiting 15:43:43 (4932): No heartbeat from core client for 30 sec - exiting 15:43:44 (4932): No heartbeat from core client for 30 sec - exiting 15:43:45 (4932): No heartbeat from core client for 30 sec - exiting 15:43:46 (4932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:30:32 (6576): No heartbeat from core client for 30 sec - exiting 04:30:33 (6576): No heartbeat from core client for 30 sec - exiting 04:30:34 (6576): No heartbeat from core client for 30 sec - exiting 04:30:35 (6576): No heartbeat from core client for 30 sec - exiting 04:30:36 (6576): No heartbeat from core client for 30 sec - exiting 04:30:37 (6576): No heartbeat from core client for 30 sec - exiting 04:30:38 (6576): No heartbeat from core client for 30 sec - exiting 04:30:39 (6576): No heartbeat from core client for 30 sec - exiting 04:30:40 (6576): No heartbeat from core client for 30 sec - exiting 04:30:41 (6576): No heartbeat from core client for 30 sec - exiting 04:30:42 (6576): No heartbeat from core client for 30 sec - exiting 04:30:43 (6576): No heartbeat from core client for 30 sec - exiting 04:30:44 (6576): No heartbeat from core client for 30 sec - exiting 04:30:45 (6576): No heartbeat from core client for 30 sec - exiting 04:30:46 (6576): No heartbeat from core client for 30 sec - exiting 04:30:47 (6576): No heartbeat from core client for 30 sec - exiting 04:30:48 (6576): No heartbeat from core client for 30 sec - exiting 04:30:49 (6576): No heartbeat from core client for 30 sec - exiting 04:30:50 (6576): No heartbeat from core client for 30 sec - exiting 04:30:51 (6576): No heartbeat from core client for 30 sec - exiting 04:30:52 (6576): No heartbeat from core client for 30 sec - exiting 04:30:53 (6576): No heartbeat from core client for 30 sec - exiting 04:30:54 (6576): No heartbeat from core client for 30 sec - exiting 04:30:55 (6576): No heartbeat from core client for 30 sec - exiting 04:30:56 (6576): No heartbeat from core client for 30 sec - exiting 04:30:57 (6576): No heartbeat from core client for 30 sec - exiting 04:30:58 (6576): No heartbeat from core client for 30 sec - exiting 04:30:59 (6576): No heartbeat from core client for 30 sec - exiting 04:31:00 (6576): No heartbeat from core client for 30 sec - exiting 04:31:01 (6576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:31:02 (6576): No heartbeat from core client for 30 sec - exiting 04:31:03 (6576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:18:14 (75016): No heartbeat from core client for 30 sec - exiting 10:18:15 (75016): No heartbeat from core client for 30 sec - exiting 10:18:16 (75016): No heartbeat from core client for 30 sec - exiting 10:18:17 (75016): No heartbeat from core client for 30 sec - exiting 10:18:18 (75016): No heartbeat from core client for 30 sec - exiting 10:18:19 (75016): No heartbeat from core client for 30 sec - exiting 10:18:20 (75016): No heartbeat from core client for 30 sec - exiting 10:18:21 (75016): No heartbeat from core client for 30 sec - exiting 10:18:22 (75016): No heartbeat from core client for 30 sec - exiting 10:18:23 (75016): No heartbeat from core client for 30 sec - exiting 10:18:24 (75016): No heartbeat from core client for 30 sec - exiting 10:18:25 (75016): No heartbeat from core client for 30 sec - exiting 10:18:26 (75016): No heartbeat from core client for 30 sec - exiting 10:18:27 (75016): No heartbeat from core client for 30 sec - exiting 10:18:28 (75016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:16:10 (11188): No heartbeat from core client for 30 sec - exiting 15:16:11 (11188): No heartbeat from core client for 30 sec - exiting 15:16:12 (11188): No heartbeat from core client for 30 sec - exiting 15:16:13 (11188): No heartbeat from core client for 30 sec - exiting 15:16:14 (11188): No heartbeat from core client for 30 sec - exiting 15:16:15 (11188): No heartbeat from core client for 30 sec - exiting 15:16:16 (11188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=60312, iMonCtr=1 Model crash detected, will try to restart... 12:54:33 (24904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:03:24 (85112): No heartbeat from core client for 30 sec - exiting 06:03:25 (85112): No heartbeat from core client for 30 sec - exiting 06:03:26 (85112): No heartbeat from core client for 30 sec - exiting 06:03:27 (85112): No heartbeat from core client for 30 sec - exiting 06:03:28 (85112): No heartbeat from core client for 30 sec - exiting 06:03:29 (85112): No heartbeat from core client for 30 sec - exiting 06:03:30 (85112): No heartbeat from core client for 30 sec - exiting 06:03:31 (85112): No heartbeat from core client for 30 sec - exiting 06:03:32 (85112): No heartbeat from core client for 30 sec - exiting 06:03:33 (85112): No heartbeat from core client for 30 sec - exiting 06:03:34 (85112): No heartbeat from core client for 30 sec - exiting 06:03:35 (85112): No heartbeat from core client for 30 sec - exiting 06:03:36 (85112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Apr 2013 10:23:27	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	1,036,800	1,335,141	1.2878
25 Apr 2013 19:10:59	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	1,010,880	1,301,653	1.2876
25 Apr 2013 01:39:37	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	984,960	1,268,082	1.2874
23 Apr 2013 16:41:10	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	959,040	1,234,564	1.2873
22 Apr 2013 11:04:40	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	933,120	1,202,158	1.2883
21 Apr 2013 13:50:10	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	907,200	1,168,976	1.2886
20 Apr 2013 12:30:12	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	881,280	1,135,740	1.2887
19 Apr 2013 18:02:07	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	855,360	1,103,110	1.2896
18 Apr 2013 18:10:02	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	829,440	1,068,923	1.2887
17 Apr 2013 23:26:25	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	803,520	1,034,916	1.2880
16 Apr 2013 20:57:00	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	777,600	1,000,840	1.2871
16 Apr 2013 04:33:21	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	751,680	966,947	1.2864
15 Apr 2013 15:20:10	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	725,760	933,269	1.2859
14 Apr 2013 14:55:34	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	699,840	899,632	1.2855
13 Apr 2013 21:47:20	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	673,920	865,939	1.2849
13 Apr 2013 04:42:06	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	648,000	832,453	1.2846
12 Apr 2013 06:43:37	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	622,080	798,322	1.2833
11 Apr 2013 10:59:42	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	596,160	764,371	1.2822
10 Apr 2013 17:11:20	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	570,240	730,632	1.2813
09 Apr 2013 20:07:02	1217831	15671339	hadcm3n_zmnq_1920_40_008316927_3	544,320	697,195	1.2809