Task 14985814

Name	hadcm3n_003y_1940_40_008085848_2
Workunit	8240962
Created	23 Jul 2012, 21:28:58 UTC
Sent	23 Jul 2012, 21:29:20 UTC
Report deadline	23 Oct 2012, 4:56:31 UTC
Received	17 Sep 2012, 9:35:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1098636
Run time	9 days 19 hours 55 min 26 sec
CPU time	9 days 18 hours 30 min 42 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.63 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5492, selfPID=5492, iMonCtr=1 CPDN Monitor - Quit request from BOINC... 07:58:10 (4480): No heartbeat from core client for 30 sec - exiting 07:58:11 (4480): No heartbeat from core client for 30 sec - exiting 07:58:12 (4480): No heartbeat from core client for 30 sec - exiting 07:58:13 (4480): No heartbeat from core client for 30 sec - exiting 07:58:14 (4480): No heartbeat from core client for 30 sec - exiting 07:58:15 (4480): No heartbeat from core client for 30 sec - exiting 07:58:16 (4480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:14:45 (6328): No heartbeat from core client for 30 sec - exiting 19:14:46 (6328): No heartbeat from core client for 30 sec - exiting 19:14:47 (6328): No heartbeat from core client for 30 sec - exiting 19:14:48 (6328): No heartbeat from core client for 30 sec - exiting 19:14:49 (6328): No heartbeat from core client for 30 sec - exiting 19:14:50 (6328): No heartbeat from core client for 30 sec - exiting 19:14:51 (6328): No heartbeat from core client for 30 sec - exiting 19:14:52 (6328): No heartbeat from core client for 30 sec - exiting 19:14:53 (6328): No heartbeat from core client for 30 sec - exiting 19:14:54 (6328): No heartbeat from core client for 30 sec - exiting 19:14:55 (6328): No heartbeat from core client for 30 sec - exiting 19:14:56 (6328): No heartbeat from core client for 30 sec - exiting 19:14:57 (6328): No heartbeat from core client for 30 sec - exiting 19:14:58 (6328): No heartbeat from core client for 30 sec - exiting 19:14:59 (6328): No heartbeat from core client for 30 sec - exiting 19:15:00 (6328): No heartbeat from core client for 30 sec - exiting 19:15:01 (6328): No heartbeat from core client for 30 sec - exiting 19:15:02 (6328): No heartbeat from core client for 30 sec - exiting 19:15:03 (6328): No heartbeat from core client for 30 sec - exiting 19:15:04 (6328): No heartbeat from core client for 30 sec - exiting 19:15:05 (6328): No heartbeat from core client for 30 sec - exiting 19:15:06 (6328): No heartbeat from core client for 30 sec - exiting 19:15:07 (6328): No heartbeat from core client for 30 sec - exiting 19:15:08 (6328): No heartbeat from core client for 30 sec - exiting 19:15:09 (6328): No heartbeat from core client for 30 sec - exiting 19:15:10 (6328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:51:29 (1772): No heartbeat from core client for 30 sec - exiting 06:51:30 (1772): No heartbeat from core client for 30 sec - exiting 06:51:31 (1772): No heartbeat from core client for 30 sec - exiting 06:51:32 (1772): No heartbeat from core client for 30 sec - exiting 06:51:33 (1772): No heartbeat from core client for 30 sec - exiting 06:51:34 (1772): No heartbeat from core client for 30 sec - exiting 06:51:35 (1772): No heartbeat from core client for 30 sec - exiting 06:51:36 (1772): No heartbeat from core client for 30 sec - exiting 06:51:37 (1772): No heartbeat from core client for 30 sec - exiting 06:51:38 (1772): No heartbeat from core client for 30 sec - exiting 06:51:39 (1772): No heartbeat from core client for 30 sec - exiting 06:51:40 (1772): No heartbeat from core client for 30 sec - exiting 06:51:41 (1772): No heartbeat from core client for 30 sec - exiting 06:51:42 (1772): No heartbeat from core client for 30 sec - exiting 06:51:43 (1772): No heartbeat from core client for 30 sec - exiting 06:51:44 (1772): No heartbeat from core client for 30 sec - exiting 06:51:45 (1772): No heartbeat from core client for 30 sec - exiting 06:51:46 (1772): No heartbeat from core client for 30 sec - exiting 06:51:47 (1772): No heartbeat from core client for 30 sec - exiting 06:51:48 (1772): No heartbeat from core client for 30 sec - exiting 06:51:49 (1772): No heartbeat from core client for 30 sec - exiting 06:51:50 (1772): No heartbeat from core client for 30 sec - exiting 06:51:51 (1772): No heartbeat from core client for 30 sec - exiting 06:51:52 (1772): No heartbeat from core client for 30 sec - exiting 06:51:53 (1772): No heartbeat from core client for 30 sec - exiting 06:51:54 (1772): No heartbeat from core client for 30 sec - exiting 06:51:55 (1772): No heartbeat from core client for 30 sec - exiting 06:51:56 (1772): No heartbeat from core client for 30 sec - exiting 06:51:57 (1772): No heartbeat from core client for 30 sec - exiting 06:51:58 (1772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:51:59 (1772): No heartbeat from core client for 30 sec - exiting 06:52:00 (1772): No heartbeat from core client for 30 sec - exiting 06:52:01 (1772): No heartbeat from core client for 30 sec - exiting 06:52:02 (1772): No heartbeat from core client for 30 sec - exiting 06:52:03 (1772): No heartbeat from core client for 30 sec - exiting 06:52:04 (1772): No heartbeat from core client for 30 sec - exiting 06:52:05 (1772): No heartbeat from core client for 30 sec - exiting 06:52:06 (1772): No heartbeat from core client for 30 sec - exiting 06:52:07 (1772): No heartbeat from core client for 30 sec - exiting 06:52:08 (1772): No heartbeat from core client for 30 sec - exiting 18:15:41 (6848): No heartbeat from core client for 30 sec - exiting 18:15:42 (6848): No heartbeat from core client for 30 sec - exiting 18:15:43 (6848): No heartbeat from core client for 30 sec - exiting 18:15:44 (6848): No heartbeat from core client for 30 sec - exiting 18:15:45 (6848): No heartbeat from core client for 30 sec - exiting 18:15:46 (6848): No heartbeat from core client for 30 sec - exiting 18:15:47 (6848): No heartbeat from core client for 30 sec - exiting 18:15:48 (6848): No heartbeat from core client for 30 sec - exiting 18:15:49 (6848): No heartbeat from core client for 30 sec - exiting 18:15:50 (6848): No heartbeat from core client for 30 sec - exiting 18:15:51 (6848): No heartbeat from core client for 30 sec - exiting 18:15:52 (6848): No heartbeat from core client for 30 sec - exiting 18:15:53 (6848): No heartbeat from core client for 30 sec - exiting 18:15:54 (6848): No heartbeat from core client for 30 sec - exiting 18:15:55 (6848): No heartbeat from core client for 30 sec - exiting 18:15:56 (6848): No heartbeat from core client for 30 sec - exiting 18:15:57 (6848): No heartbeat from core client for 30 sec - exiting 18:15:58 (6848): No heartbeat from core client for 30 sec - exiting 18:15:59 (6848): No heartbeat from core client for 30 sec - exiting 18:16:00 (6848): No heartbeat from core client for 30 sec - exiting 18:16:01 (6848): No heartbeat from core client for 30 sec - exiting 18:16:02 (6848): No heartbeat from core client for 30 sec - exiting 18:16:03 (6848): No heartbeat from core client for 30 sec - exiting 18:16:04 (6848): No heartbeat from core client for 30 sec - exiting 18:16:05 (6848): No heartbeat from core client for 30 sec - exiting 18:16:06 (6848): No heartbeat from core client for 30 sec - exiting 18:16:07 (6848): No heartbeat from core client for 30 sec - exiting 18:16:08 (6848): No heartbeat from core client for 30 sec - exiting 18:16:09 (6848): No heartbeat from core client for 30 sec - exiting 18:16:10 (6848): No heartbeat from core client for 30 sec - exiting 18:16:11 (6848): No heartbeat from core client for 30 sec - exiting 18:16:12 (6848): No heartbeat from core client for 30 sec - exiting 18:16:13 (6848): No heartbeat from core client for 30 sec - exiting 18:16:14 (6848): No heartbeat from core client for 30 sec - exiting 18:16:15 (6848): No heartbeat from core client for 30 sec - exiting 18:16:16 (6848): No heartbeat from core client for 30 sec - exiting 18:16:17 (6848): No heartbeat from core client for 30 sec - exiting 18:16:18 (6848): No heartbeat from core client for 30 sec - exiting 18:16:19 (6848): No heartbeat from core client for 30 sec - exiting 18:16:20 (6848): No heartbeat from core client for 30 sec - exiting 18:16:21 (6848): No heartbeat from core client for 30 sec - exiting 18:16:22 (6848): No heartbeat from core client for 30 sec - exiting 18:16:23 (6848): No heartbeat from core client for 30 sec - exiting 18:16:24 (6848): No heartbeat from core client for 30 sec - exiting 18:16:25 (6848): No heartbeat from core client for 30 sec - exiting 18:16:26 (6848): No heartbeat from core client for 30 sec - exiting 18:16:27 (6848): No heartbeat from core client for 30 sec - exiting 18:16:28 (6848): No heartbeat from core client for 30 sec - exiting 18:16:29 (6848): No heartbeat from core client for 30 sec - exiting 18:16:30 (6848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5084, iMonCtr=1 Model crash detected, will try to restart... 07:53:34 (4624): No heartbeat from core client for 30 sec - exiting 07:53:36 (4624): No heartbeat from core client for 30 sec - exiting 07:53:37 (4624): No heartbeat from core client for 30 sec - exiting 07:53:38 (4624): No heartbeat from core client for 30 sec - exiting 07:53:39 (4624): No heartbeat from core client for 30 sec - exiting 07:53:40 (4624): No heartbeat from core client for 30 sec - exiting 07:53:41 (4624): No heartbeat from core client for 30 sec - exiting 07:53:42 (4624): No heartbeat from core client for 30 sec - exiting 07:53:43 (4624): No heartbeat from core client for 30 sec - exiting 07:53:44 (4624): No heartbeat from core client for 30 sec - exiting 07:53:45 (4624): No heartbeat from core client for 30 sec - exiting 07:53:46 (4624): No heartbeat from core client for 30 sec - exiting 07:53:47 (4624): No heartbeat from core client for 30 sec - exiting 07:53:48 (4624): No heartbeat from core client for 30 sec - exiting 07:53:49 (4624): No heartbeat from core client for 30 sec - exiting 07:53:50 (4624): No heartbeat from core client for 30 sec - exiting 07:53:51 (4624): No heartbeat from core client for 30 sec - exiting 07:53:52 (4624): No heartbeat from core client for 30 sec - exiting 07:53:53 (4624): No heartbeat from core client for 30 sec - exiting 07:53:54 (4624): No heartbeat from core client for 30 sec - exiting 07:53:55 (4624): No heartbeat from core client for 30 sec - exiting 07:53:56 (4624): No heartbeat from core client for 30 sec - exiting 07:53:57 (4624): No heartbeat from core client for 30 sec - exiting 07:53:58 (4624): No heartbeat from core client for 30 sec - exiting 07:53:59 (4624): No heartbeat from core client for 30 sec - exiting 07:54:00 (4624): No heartbeat from core client for 30 sec - exiting 07:54:01 (4624): No heartbeat from core client for 30 sec - exiting 07:54:02 (4624): No heartbeat from core client for 30 sec - exiting 07:54:03 (4624): No heartbeat from core client for 30 sec - exiting 07:54:04 (4624): No heartbeat from core client for 30 sec - exiting 07:54:05 (4624): No heartbeat from core client for 30 sec - exiting 07:54:06 (4624): No heartbeat from core client for 30 sec - exiting 07:54:07 (4624): No heartbeat from core client for 30 sec - exiting 07:54:08 (4624): No heartbeat from core client for 30 sec - exiting 07:54:09 (4624): No heartbeat from core client for 30 sec - exiting 07:54:10 (4624): No heartbeat from core client for 30 sec - exiting 07:54:11 (4624): No heartbeat from core client for 30 sec - exiting 07:54:12 (4624): No heartbeat from core client for 30 sec - exiting 07:54:13 (4624): No heartbeat from core client for 30 sec - exiting 07:54:14 (4624): No heartbeat from core client for 30 sec - exiting 07:54:15 (4624): No heartbeat from core client for 30 sec - exiting 07:54:16 (4624): No heartbeat from core client for 30 sec - exiting 07:54:17 (4624): No heartbeat from core client for 30 sec - exiting 07:54:18 (4624): No heartbeat from core client for 30 sec - exiting 07:54:19 (4624): No heartbeat from core client for 30 sec - exiting 07:54:20 (4624): No heartbeat from core client for 30 sec - exiting 07:54:21 (4624): No heartbeat from core client for 30 sec - exiting 07:54:22 (4624): No heartbeat from core client for 30 sec - exiting 07:54:23 (4624): No heartbeat from core client for 30 sec - exiting 07:54:24 (4624): No heartbeat from core client for 30 sec - exiting 07:54:25 (4624): No heartbeat from core client for 30 sec - exiting 07:54:26 (4624): No heartbeat from core client for 30 sec - exiting 07:54:27 (4624): No heartbeat from core client for 30 sec - exiting 07:54:28 (4624): No heartbeat from core client for 30 sec - exiting 07:54:29 (4624): No heartbeat from core client for 30 sec - exiting 07:54:30 (4624): No heartbeat from core client for 30 sec - exiting 07:54:31 (4624): No heartbeat from core client for 30 sec - exiting 07:54:32 (4624): No heartbeat from core client for 30 sec - exiting 07:54:33 (4624): No heartbeat from core client for 30 sec - exiting 07:54:34 (4624): No heartbeat from core client for 30 sec - exiting 07:54:35 (4624): No heartbeat from core client for 30 sec - exiting 07:54:36 (4624): No heartbeat from core client for 30 sec - exiting 07:54:37 (4624): No heartbeat from core client for 30 sec - exiting 07:54:38 (4624): No heartbeat from core client for 30 sec - exiting 07:54:39 (4624): No heartbeat from core client for 30 sec - exiting 07:54:40 (4624): No heartbeat from core client for 30 sec - exiting 07:54:41 (4624): No heartbeat from core client for 30 sec - exiting 07:54:42 (4624): No heartbeat from core client for 30 sec - exiting 07:54:43 (4624): No heartbeat from core client for 30 sec - exiting 07:54:44 (4624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:23:17 (4008): No heartbeat from core client for 30 sec - exiting 09:23:18 (4008): No heartbeat from core client for 30 sec - exiting 09:23:19 (4008): No heartbeat from core client for 30 sec - exiting 09:23:20 (4008): No heartbeat from core client for 30 sec - exiting 09:23:51 (4008): No heartbeat from core client for 30 sec - exiting 09:23:52 (4008): No heartbeat from core client for 30 sec - exiting 09:23:53 (4008): No heartbeat from core client for 30 sec - exiting 09:23:54 (4008): No heartbeat from core client for 30 sec - exiting 09:23:55 (4008): No heartbeat from core client for 30 sec - exiting 09:23:56 (4008): No heartbeat from core client for 30 sec - exiting 09:23:57 (4008): No heartbeat from core client for 30 sec - exiting 09:23:58 (4008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:23:59 (4008): No heartbeat from core client for 30 sec - exiting 09:24:00 (4008): No heartbeat from core client for 30 sec - exiting 09:24:01 (4008): No heartbeat from core client for 30 sec - exiting 09:24:02 (4008): No heartbeat from core client for 30 sec - exiting 09:24:03 (4008): No heartbeat from core client for 30 sec - exiting 09:24:04 (4008): No heartbeat from core client for 30 sec - exiting 09:24:05 (4008): No heartbeat from core client for 30 sec - exiting 09:24:06 (4008): No heartbeat from core client for 30 sec - exiting 09:24:07 (4008): No heartbeat from core client for 30 sec - exiting 09:24:08 (4008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:40:55 (3444): No heartbeat from core client for 30 sec - exiting 13:40:56 (3444): No heartbeat from core client for 30 sec - exiting 13:40:57 (3444): No heartbeat from core client for 30 sec - exiting 13:40:58 (3444): No heartbeat from core client for 30 sec - exiting 13:40:59 (3444): No heartbeat from core client for 30 sec - exiting 13:41:00 (3444): No heartbeat from core client for 30 sec - exiting 13:41:01 (3444): No heartbeat from core client for 30 sec - exiting 13:41:02 (3444): No heartbeat from core client for 30 sec - exiting 13:41:03 (3444): No heartbeat from core client for 30 sec - exiting 13:41:04 (3444): No heartbeat from core client for 30 sec - exiting 13:41:05 (3444): No heartbeat from core client for 30 sec - exiting 13:41:06 (3444): No heartbeat from core client for 30 sec - exiting 13:41:07 (3444): No heartbeat from core client for 30 sec - exiting 13:41:08 (3444): No heartbeat from core client for 30 sec - exiting 13:41:09 (3444): No heartbeat from core client for 30 sec - exiting 13:41:10 (3444): No heartbeat from core client for 30 sec - exiting 13:41:11 (3444): No heartbeat from core client for 30 sec - exiting 13:41:12 (3444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/003yko.pjf0c10 Error converting file to netcdf: dataout/003yko.pif0c10 Error converting file to netcdf: dataout/003yko.pff0c10 Error converting file to netcdf: dataout/003yko.pcf0c10 Error converting file to netcdf: dataout/003yko.pbf0c10 Error converting file to netcdf: dataout/003yko.paf0c10 Error converting file to netcdf: dataout/003yka.phf0c10 Error converting file to netcdf: dataout/003yka.pgf0c10 Error converting file to netcdf: dataout/003yka.pef0c10 Error converting file to netcdf: dataout/003yka.pdf0c10 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:56:37 (5052): No heartbeat from core client for 30 sec - exiting 20:56:38 (5052): No heartbeat from core client for 30 sec - exiting 20:56:39 (5052): No heartbeat from core client for 30 sec - exiting 20:56:40 (5052): No heartbeat from core client for 30 sec - exiting 20:56:41 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:56:42 (5052): No heartbeat from core client for 30 sec - exiting 20:56:43 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... 20:30:34 (6132): No heartbeat from core client for 30 sec - exiting 20:30:35 (6132): No heartbeat from core client for 30 sec - exiting 20:30:36 (6132): No heartbeat from core client for 30 sec - exiting 20:30:37 (6132): No heartbeat from core client for 30 sec - exiting 20:30:38 (6132): No heartbeat from core client for 30 sec - exiting 20:30:39 (6132): No heartbeat from core client for 30 sec - exiting 20:30:40 (6132): No heartbeat from core client for 30 sec - exiting 20:30:41 (6132): No heartbeat from core client for 30 sec - exiting 20:30:42 (6132): No heartbeat from core client for 30 sec - exiting 20:30:43 (6132): No heartbeat from core client for 30 sec - exiting 20:30:44 (6132): No heartbeat from core client for 30 sec - exiting 20:30:45 (6132): No heartbeat from core client for 30 sec - exiting 20:30:46 (6132): No heartbeat from core client for 30 sec - exiting 20:30:47 (6132): No heartbeat from core client for 30 sec - exiting 20:30:48 (6132): No heartbeat from core client for 30 sec - exiting 20:30:49 (6132): No heartbeat from core client for 30 sec - exiting 20:30:50 (6132): No heartbeat from core client for 30 sec - exiting 20:30:51 (6132): No heartbeat from core client for 30 sec - exiting 20:30:52 (6132): No heartbeat from core client for 30 sec - exiting 20:30:53 (6132): No heartbeat from core client for 30 sec - exiting 20:30:54 (6132): No heartbeat from core client for 30 sec - exiting 20:30:55 (6132): No heartbeat from core client for 30 sec - exiting 20:30:56 (6132): No heartbeat from core client for 30 sec - exiting 20:30:57 (6132): No heartbeat from core client for 30 sec - exiting 20:30:58 (6132): No heartbeat from core client for 30 sec - exiting 20:30:59 (6132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:38:44 (6948): No heartbeat from core client for 30 sec - exiting 07:38:45 (6948): No heartbeat from core client for 30 sec - exiting 07:38:46 (6948): No heartbeat from core client for 30 sec - exiting 07:38:47 (6948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6508, iMonCtr=1 Model crash detected, will try to restart... 18:47:04 (3824): No heartbeat from core client for 30 sec - exiting 18:47:05 (3824): No heartbeat from core client for 30 sec - exiting 18:47:06 (3824): No heartbeat from core client for 30 sec - exiting 18:47:07 (3824): No heartbeat from core client for 30 sec - exiting 18:47:08 (3824): No heartbeat from core client for 30 sec - exiting 18:47:09 (3824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:15:55 (7388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:43:28 (4304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:55:07 (6452): No heartbeat from core client for 30 sec - exiting 18:55:08 (6452): No heartbeat from core client for 30 sec - exiting 18:55:09 (6452): No heartbeat from core client for 30 sec - exiting 18:55:10 (6452): No heartbeat from core client for 30 sec - exiting 18:55:11 (6452): No heartbeat from core client for 30 sec - exiting 18:55:12 (6452): No heartbeat from core client for 30 sec - exiting 18:55:13 (6452): No heartbeat from core client for 30 sec - exiting 18:55:14 (6452): No heartbeat from core client for 30 sec - exiting 18:55:15 (6452): No heartbeat from core client for 30 sec - exiting 18:55:16 (6452): No heartbeat from core client for 30 sec - exiting 18:55:17 (6452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:20:12 (5084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:20:13 (5084): No heartbeat from core client for 30 sec - exiting 16:20:14 (5084): No heartbeat from core client for 30 sec - exiting 16:20:15 (5084): No heartbeat from core client for 30 sec - exiting 16:20:16 (5084): No heartbeat from core client for 30 sec - exiting 16:20:17 (5084): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:54:25 (6880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:24:34 (4696): No heartbeat from core client for 30 sec - exiting 14:24:35 (4696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:27:45 (5372): No heartbeat from core client for 30 sec - exiting 09:27:46 (5372): No heartbeat from core client for 30 sec - exiting 09:27:47 (5372): No heartbeat from core client for 30 sec - exiting 09:27:48 (5372): No heartbeat from core client for 30 sec - exiting 09:27:49 (5372): No heartbeat from core client for 30 sec - exiting 09:27:50 (5372): No heartbeat from core client for 30 sec - exiting 09:27:51 (5372): No heartbeat from core client for 30 sec - exiting 09:27:52 (5372): No heartbeat from core client for 30 sec - exiting 09:27:53 (5372): No heartbeat from core client for 30 sec - exiting 09:27:54 (5372): No heartbeat from core client for 30 sec - exiting 09:27:55 (5372): No heartbeat from core client for 30 sec - exiting 09:27:56 (5372): No heartbeat from core client for 30 sec - exiting 09:27:57 (5372): No heartbeat from core client for 30 sec - exiting 09:27:58 (5372): No heartbeat from core client for 30 sec - exiting 09:27:59 (5372): No heartbeat from core client for 30 sec - exiting 09:28:00 (5372): No heartbeat from core client for 30 sec - exiting 09:28:01 (5372): No heartbeat from core client for 30 sec - exiting 09:28:02 (5372): No heartbeat from core client for 30 sec - exiting 09:28:03 (5372): No heartbeat from core client for 30 sec - exiting 09:28:04 (5372): No heartbeat from core client for 30 sec - exiting 09:28:05 (5372): No heartbeat from core client for 30 sec - exiting 09:28:06 (5372): No heartbeat from core client for 30 sec - exiting 09:28:07 (5372): No heartbeat from core client for 30 sec - exiting 09:28:08 (5372): No heartbeat from core client for 30 sec - exiting 09:28:09 (5372): No heartbeat from core client for 30 sec - exiting cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_003y_1940_40_008085848/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Sep 2012 09:40:25	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	518,400	844,213	1.6285
16 Sep 2012 11:45:14	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	492,480	804,994	1.6346
15 Sep 2012 14:43:54	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	466,560	765,678	1.6411
14 Sep 2012 14:23:46	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	440,640	720,991	1.6362
12 Sep 2012 10:29:53	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	414,720	679,538	1.6385
10 Sep 2012 20:32:07	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	388,800	638,019	1.6410
09 Sep 2012 16:27:12	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	362,880	598,551	1.6494
07 Sep 2012 12:15:39	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	336,960	556,046	1.6502
04 Sep 2012 21:52:37	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	311,040	513,412	1.6506
02 Sep 2012 14:56:56	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	285,120	472,550	1.6574
01 Sep 2012 12:46:13	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	259,200	430,447	1.6607
30 Aug 2012 22:58:04	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	233,280	388,818	1.6667
29 Aug 2012 21:07:53	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	207,360	347,748	1.6770
28 Aug 2012 16:59:45	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	181,440	306,026	1.6867
27 Aug 2012 14:02:32	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	155,520	266,005	1.7104
26 Aug 2012 14:36:20	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	129,600	222,641	1.7179
25 Aug 2012 15:41:06	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	103,680	178,180	1.7186
24 Aug 2012 17:55:24	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	77,760	134,672	1.7319
22 Aug 2012 14:12:39	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	51,840	89,747	1.7312
21 Aug 2012 19:41:56	1098636	14985814	hadcm3n_003y_1940_40_008085848_2	25,920	43,793	1.6895