Task 15281798

Name	hadcm3n_zg9y_1880_40_008201307_1
Workunit	8356431
Created	13 Sep 2012, 12:23:50 UTC
Sent	13 Sep 2012, 16:24:33 UTC
Report deadline	13 Dec 2012, 23:51:44 UTC
Received	28 Oct 2012, 22:23:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1116570
Run time	12 days 9 hours 57 min 14 sec
CPU time	11 days 19 hours 36 min 3 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.65 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5608, iMonCtr=1 Model crash detected, will try to restart... 17:51:55 (4632): No heartbeat from core client for 30 sec - exiting 17:51:56 (4632): No heartbeat from core client for 30 sec - exiting 17:51:57 (4632): No heartbeat from core client for 30 sec - exiting 17:51:58 (4632): No heartbeat from core client for 30 sec - exiting 17:51:59 (4632): No heartbeat from core client for 30 sec - exiting 17:52:00 (4632): No heartbeat from core client for 30 sec - exiting 17:52:01 (4632): No heartbeat from core client for 30 sec - exiting 17:52:02 (4632): No heartbeat from core client for 30 sec - exiting 17:52:03 (4632): No heartbeat from core client for 30 sec - exiting 17:52:04 (4632): No heartbeat from core client for 30 sec - exiting 17:52:06 (4632): No heartbeat from core client for 30 sec - exiting 17:52:07 (4632): No heartbeat from core client for 30 sec - exiting 17:52:08 (4632): No heartbeat from core client for 30 sec - exiting 17:52:09 (4632): No heartbeat from core client for 30 sec - exiting 17:52:10 (4632): No heartbeat from core client for 30 sec - exiting 17:52:11 (4632): No heartbeat from core client for 30 sec - exiting 17:52:12 (4632): No heartbeat from core client for 30 sec - exiting 17:52:13 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:13:45 (5892): No heartbeat from core client for 30 sec - exiting 08:13:46 (5892): No heartbeat from core client for 30 sec - exiting 08:13:48 (5892): No heartbeat from core client for 30 sec - exiting 08:13:49 (5892): No heartbeat from core client for 30 sec - exiting 08:13:50 (5892): No heartbeat from core client for 30 sec - exiting 08:13:51 (5892): No heartbeat from core client for 30 sec - exiting 08:13:52 (5892): No heartbeat from core client for 30 sec - exiting 08:13:53 (5892): No heartbeat from core client for 30 sec - exiting 08:13:54 (5892): No heartbeat from core client for 30 sec - exiting 08:13:55 (5892): No heartbeat from core client for 30 sec - exiting 08:13:56 (5892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:50 (3604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1128, iMonCtr=1 Model crash detected, will try to restart... 08:14:02 (5188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:02:08 (4140): No heartbeat from core client for 30 sec - exiting 10:02:09 (4140): No heartbeat from core client for 30 sec - exiting 10:02:10 (4140): No heartbeat from core client for 30 sec - exiting 10:02:11 (4140): No heartbeat from core client for 30 sec - exiting 10:02:12 (4140): No heartbeat from core client for 30 sec - exiting 10:02:13 (4140): No heartbeat from core client for 30 sec - exiting 10:02:14 (4140): No heartbeat from core client for 30 sec - exiting 10:02:15 (4140): No heartbeat from core client for 30 sec - exiting 10:02:16 (4140): No heartbeat from core client for 30 sec - exiting 10:02:17 (4140): No heartbeat from core client for 30 sec - exiting 10:02:19 (4140): No heartbeat from core client for 30 sec - exiting 10:02:20 (4140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:08:42 (4108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:10:26 (5564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:10:27 (5564): No heartbeat from core client for 30 sec - exiting 18:10:28 (5564): No heartbeat from core client for 30 sec - exiting 18:10:29 (5564): No heartbeat from core client for 30 sec - exiting 18:10:30 (5564): No heartbeat from core client for 30 sec - exiting 18:10:31 (5564): No heartbeat from core client for 30 sec - exiting 18:10:32 (5564): No heartbeat from core client for 30 sec - exiting 18:10:33 (5564): No heartbeat from core client for 30 sec - exiting 18:10:34 (5564): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:55:26 (5736): No heartbeat from core client for 30 sec - exiting 17:55:28 (5736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:56:16 (4256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:57:09 (5532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:55:07 (4028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:55:09 (4028): No heartbeat from core client for 30 sec - exiting 18:55:10 (4028): No heartbeat from core client for 30 sec - exiting 18:55:11 (4028): No heartbeat from core client for 30 sec - exiting 18:56:01 (2764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:34 (1408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zg9yko.pj83c10 Error converting file to netcdf: dataout/zg9yko.pi83c10 Error converting file to netcdf: dataout/zg9yko.pf83c10 Error converting file to netcdf: dataout/zg9yka.ph83c10 Error converting file to netcdf: dataout/zg9yka.pg83c10 Error converting file to netcdf: dataout/zg9yka.pe83c10 Error converting file to netcdf: dataout/zg9yka.pd83c10 18:03:44 (2224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zg9yko.pj83c10 Error converting file to netcdf: dataout/zg9yko.pi83c10 Error converting file to netcdf: dataout/zg9yko.pf83c10 Error converting file to netcdf: dataout/zg9yka.ph83c10 Error converting file to netcdf: dataout/zg9yka.pg83c10 Error converting file to netcdf: dataout/zg9yka.pe83c10 Error converting file to netcdf: dataout/zg9yka.pd83c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5784, iMonCtr=1 Model crash detected, will try to restart... 17:54:53 (2304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:34 (3540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3508, iMonCtr=1 Model crash detected, will try to restart... 21:08:22 (4436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:08:58 (7112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:04:25 (1280): No heartbeat from core client for 30 sec - exiting 18:04:26 (1280): No heartbeat from core client for 30 sec - exiting 18:04:27 (1280): No heartbeat from core client for 30 sec - exiting 18:04:28 (1280): No heartbeat from core client for 30 sec - exiting 18:04:30 (1280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:15 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:23:20 (4724): No heartbeat from core client for 30 sec - exiting 17:23:21 (4724): No heartbeat from core client for 30 sec - exiting 17:23:22 (4724): No heartbeat from core client for 30 sec - exiting 17:23:24 (4724): No heartbeat from core client for 30 sec - exiting 17:23:25 (4724): No heartbeat from core client for 30 sec - exiting 17:23:26 (4724): No heartbeat from core client for 30 sec - exiting 17:23:27 (4724): No heartbeat from core client for 30 sec - exiting 17:23:28 (4724): No heartbeat from core client for 30 sec - exiting 17:23:29 (4724): No heartbeat from core client for 30 sec - exiting 17:23:30 (4724): No heartbeat from core client for 30 sec - exiting 17:23:31 (4724): No heartbeat from core client for 30 sec - exiting 17:23:32 (4724): No heartbeat from core client for 30 sec - exiting 17:23:33 (4724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:34 (4724): No heartbeat from core client for 30 sec - exiting 17:23:36 (4724): No heartbeat from core client for 30 sec - exiting 17:24:17 (3620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:53:44 (3336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:53:46 (3336): No heartbeat from core client for 30 sec - exiting 17:53:47 (3336): No heartbeat from core client for 30 sec - exiting 17:53:48 (3336): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:45:24 (3644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3984, iMonCtr=1 Model crash detected, will try to restart... 18:30:44 (5876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:45 (5876): No heartbeat from core client for 30 sec - exiting 17:28:19 (3120): No heartbeat from core client for 30 sec - exiting 17:28:25 (3120): No heartbeat from core client for 30 sec - exiting 17:28:26 (3120): No heartbeat from core client for 30 sec - exiting 17:28:27 (3120): No heartbeat from core client for 30 sec - exiting 17:28:28 (3120): No heartbeat from core client for 30 sec - exiting 17:28:30 (3120): No heartbeat from core client for 30 sec - exiting 17:28:31 (3120): No heartbeat from core client for 30 sec - exiting 17:28:32 (3120): No heartbeat from core client for 30 sec - exiting 17:28:33 (3120): No heartbeat from core client for 30 sec - exiting 17:28:34 (3120): No heartbeat from core client for 30 sec - exiting 17:28:35 (3120): No heartbeat from core client for 30 sec - exiting 17:28:36 (3120): No heartbeat from core client for 30 sec - exiting 17:28:37 (3120): No heartbeat from core client for 30 sec - exiting 17:28:38 (3120): No heartbeat from core client for 30 sec - exiting 17:28:39 (3120): No heartbeat from core client for 30 sec - exiting 17:28:40 (3120): No heartbeat from core client for 30 sec - exiting 17:28:42 (3120): No heartbeat from core client for 30 sec - exiting 17:28:43 (3120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:29:40 (4948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:32 (5992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:08 (2560): No heartbeat from core client for 30 sec - exiting 17:31:09 (2560): No heartbeat from core client for 30 sec - exiting 17:31:11 (2560): No heartbeat from core client for 30 sec - exiting 17:31:12 (2560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:22:43 (5748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:51:06 (152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:26:50 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:26:52 (5052): No heartbeat from core client for 30 sec - exiting 12:26:53 (5052): No heartbeat from core client for 30 sec - exiting 12:26:54 (5052): No heartbeat from core client for 30 sec - exiting 12:26:55 (5052): No heartbeat from core client for 30 sec - exiting 12:26:56 (5052): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:13:33 (4912): No heartbeat from core client for 30 sec - exiting 08:13:34 (4912): No heartbeat from core client for 30 sec - exiting 08:13:36 (4912): No heartbeat from core client for 30 sec - exiting 08:13:37 (4912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:38 (4912): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2212, iMonCtr=1 Model crash detected, will try to restart... 17:46:53 (5364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3596, iMonCtr=1 Model crash detected, will try to restart... 18:18:25 (6072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:34:32 (5204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:01:18 (5828): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 12:01:19 (5828): No heartbeat from core client for 30 sec - exiting 12:01:20 (5828): No heartbeat from core client for 30 sec - exiting 12:01:21 (5828): No heartbeat from core client for 30 sec - exiting 12:04:10 (13444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14564, iMonCtr=1 Model crash detected, will try to restart... 08:34:47 (4040): No heartbeat from core client for 30 sec - exiting 08:34:48 (4040): No heartbeat from core client for 30 sec - exiting 08:34:49 (4040): No heartbeat from core client for 30 sec - exiting 08:34:50 (4040): No heartbeat from core client for 30 sec - exiting 08:34:51 (4040): No heartbeat from core client for 30 sec - exiting 08:34:52 (4040): No heartbeat from core client for 30 sec - exiting 08:34:54 (4040): No heartbeat from core client for 30 sec - exiting 08:34:55 (4040): No heartbeat from core client for 30 sec - exiting 08:34:56 (4040): No heartbeat from core client for 30 sec - exiting 08:34:57 (4040): No heartbeat from core client for 30 sec - exiting 08:34:58 (4040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:14:41 (5928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:14:43 (5928): No heartbeat from core client for 30 sec - exiting 16:14:44 (5928): No heartbeat from core client for 30 sec - exiting 16:14:45 (5928): No heartbeat from core client for 30 sec - exiting 16:14:46 (5928): No heartbeat from core client for 30 sec - exiting 16:16:47 (3756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5988, iMonCtr=1 Model crash detected, will try to restart... 08:47:31 (5768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:48:20 (5172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5772, iMonCtr=1 Model crash detected, will try to restart... 17:51:03 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:16:26 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:16:27 (4360): No heartbeat from core client for 30 sec - exiting 19:16:28 (4360): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=820, iMonCtr=1 Model crash detected, will try to restart... 17:38:36 (4456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:25 (4696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:27 (4696): No heartbeat from core client for 30 sec - exiting 17:55:28 (4696): No heartbeat from core client for 30 sec - exiting 17:57:01 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6084, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6084, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6084, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6084, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6084, iMonCtr=1 Model crash detected, will try to restart... 18:06:45 (5716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4764, iMonCtr=1 Model crash detected, will try to restart... 17:58:34 (4876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:16:40 (5012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1540, iMonCtr=1 Model crash detected, will try to restart... 17:55:45 (3908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6016, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3924, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3976, iMonCtr=1 Model crash detected, will try to restart... 18:06:18 (136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:16:39 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Oct 2012 22:25:28	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	518,400	1,020,953	1.9694
27 Oct 2012 20:47:40	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	492,480	968,778	1.9671
27 Oct 2012 06:18:40	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	466,560	917,886	1.9673
23 Oct 2012 19:28:28	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	440,640	867,207	1.9681
21 Oct 2012 12:10:21	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	414,720	817,077	1.9702
20 Oct 2012 12:21:59	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	388,800	767,402	1.9738
18 Oct 2012 15:58:02	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	362,880	717,149	1.9763
10 Oct 2012 18:45:29	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	336,960	666,546	1.9781
07 Oct 2012 14:11:21	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	311,040	616,206	1.9811
06 Oct 2012 14:53:56	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	285,120	564,235	1.9789
05 Oct 2012 16:14:13	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	259,200	513,433	1.9808
04 Oct 2012 15:09:33	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	233,280	461,155	1.9768
03 Oct 2012 15:02:39	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	207,360	410,311	1.9787
01 Oct 2012 18:59:51	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	181,440	360,027	1.9843
30 Sep 2012 07:26:18	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	155,520	308,169	1.9815
29 Sep 2012 07:25:36	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	129,600	256,753	1.9811
25 Sep 2012 17:40:16	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	103,680	205,415	1.9812
19 Sep 2012 17:08:56	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	77,760	154,652	1.9888
16 Sep 2012 12:30:32	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	51,840	103,480	1.9961
15 Sep 2012 13:38:41	1116570	15281798	hadcm3n_zg9y_1880_40_008201307_1	25,920	51,694	1.9944