Task 13627204

Name	hadcm3n_o1hp_1940_40_007543576_1
Workunit	7740808
Created	10 Nov 2011, 2:08:05 UTC
Sent	16 Nov 2011, 7:02:28 UTC
Report deadline	15 Feb 2012, 14:29:39 UTC
Received	31 Jan 2012, 8:16:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1160027
Run time	17 days 21 hours 49 min 53 sec
CPU time	17 days 9 hours 27 min 18 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.67 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7008, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3156, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 08:39:22 (4524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3556, iMonCtr=1 Model crash detected, will try to restart... 08:35:59 (3528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4628, iMonCtr=1 Model crash detected, will try to restart... 08:46:54 (3120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:46:55 (3120): No heartbeat from core client for 30 sec - exiting 08:48:11 (5952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:48:50 (5680): No heartbeat from core client for 30 sec - exiting 08:48:51 (5680): No heartbeat from core client for 30 sec - exiting 08:48:52 (5680): No heartbeat from core client for 30 sec - exiting 08:48:53 (5680): No heartbeat from core client for 30 sec - exiting 08:48:54 (5680): No heartbeat from core client for 30 sec - exiting 08:48:55 (5680): No heartbeat from core client for 30 sec - exiting 08:48:56 (5680): No heartbeat from core client for 30 sec - exiting 08:48:57 (5680): No heartbeat from core client for 30 sec - exiting 08:48:58 (5680): No heartbeat from core client for 30 sec - exiting 08:48:59 (5680): No heartbeat from core client for 30 sec - exiting 08:49:00 (5680): No heartbeat from core client for 30 sec - exiting 08:49:01 (5680): No heartbeat from core client for 30 sec - exiting 08:49:02 (5680): No heartbeat from core client for 30 sec - exiting 08:49:03 (5680): No heartbeat from core client for 30 sec - exiting 08:49:04 (5680): No heartbeat from core client for 30 sec - exiting 08:49:05 (5680): No heartbeat from core client for 30 sec - exiting 08:49:06 (5680): No heartbeat from core client for 30 sec - exiting 08:49:07 (5680): No heartbeat from core client for 30 sec - exiting 08:49:08 (5680): No heartbeat from core client for 30 sec - exiting 08:49:09 (5680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:49:10 (5680): No heartbeat from core client for 30 sec - exiting 08:49:11 (5680): No heartbeat from core client for 30 sec - exiting 08:49:13 (5680): No heartbeat from core client for 30 sec - exiting 08:49:14 (5680): No heartbeat from core client for 30 sec - exiting 08:49:15 (5680): No heartbeat from core client for 30 sec - exiting 08:49:16 (5680): No heartbeat from core client for 30 sec - exiting 08:49:17 (5680): No heartbeat from core client for 30 sec - exiting 08:50:00 (1900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:53:22 (6700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4008, iMonCtr=1 Model crash detected, will try to restart... 15:32:54 (3288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4552, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2380, iMonCtr=1 Model crash detected, will try to restart... 08:32:31 (1640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3800, iMonCtr=1 Model crash detected, will try to restart... 16:42:55 (4400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:42:58 (4400): No heartbeat from core client for 30 sec - exiting 16:42:59 (4400): No heartbeat from core client for 30 sec - exiting 16:43:00 (4400): No heartbeat from core client for 30 sec - exiting 16:43:01 (4400): No heartbeat from core client for 30 sec - exiting 16:43:02 (4400): No heartbeat from core client for 30 sec - exiting 16:44:29 (8044): No heartbeat from core client for 30 sec - exiting 16:44:30 (8044): No heartbeat from core client for 30 sec - exiting 16:44:31 (8044): No heartbeat from core client for 30 sec - exiting 16:44:32 (8044): No heartbeat from core client for 30 sec - exiting 16:44:33 (8044): No heartbeat from core client for 30 sec - exiting 16:44:34 (8044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:44:35 (8044): No heartbeat from core client for 30 sec - exiting 17:12:59 (2228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:13:01 (2228): No heartbeat from core client for 30 sec - exiting 17:13:02 (2228): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4324, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2592, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2392, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3032, iMonCtr=1 Model crash detected, will try to restart... 08:44:35 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:34:40 (3172): No heartbeat from core client for 30 sec - exiting 08:34:41 (3172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:42 (3172): No heartbeat from core client for 30 sec - exiting 08:34:43 (3172): No heartbeat from core client for 30 sec - exiting 08:34:44 (3172): No heartbeat from core client for 30 sec - exiting 08:34:45 (3172): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4076, iMonCtr=1 Model crash detected, will try to restart... 08:35:41 (2976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4948, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:27:26 (44696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:27 (44696): No heartbeat from core client for 30 sec - exiting 10:27:29 (44696): No heartbeat from core client for 30 sec - exiting 10:47:29 (44480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:47:31 (44480): No heartbeat from core client for 30 sec - exiting 10:47:32 (44480): No heartbeat from core client for 30 sec - exiting 10:47:33 (44480): No heartbeat from core client for 30 sec - exiting 11:07:34 (45692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:07:36 (45692): No heartbeat from core client for 30 sec - exiting 11:07:37 (45692): No heartbeat from core client for 30 sec - exiting 11:37:39 (45932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:37:41 (45932): No heartbeat from core client for 30 sec - exiting 11:37:42 (45932): No heartbeat from core client for 30 sec - exiting 11:57:41 (45904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:57:43 (45904): No heartbeat from core client for 30 sec - exiting 11:57:44 (45904): No heartbeat from core client for 30 sec - exiting 11:57:45 (45904): No heartbeat from core client for 30 sec - exiting 12:27:43 (45768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:27:44 (45768): No heartbeat from core client for 30 sec - exiting 12:27:45 (45768): No heartbeat from core client for 30 sec - exiting 12:27:46 (45768): No heartbeat from core client for 30 sec - exiting 12:52:46 (40604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:52:48 (40604): No heartbeat from core client for 30 sec - exiting 12:52:49 (40604): No heartbeat from core client for 30 sec - exiting 13:02:47 (45808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:02:48 (45808): No heartbeat from core client for 30 sec - exiting 13:02:49 (45808): No heartbeat from core client for 30 sec - exiting 13:02:50 (45808): No heartbeat from core client for 30 sec - exiting 13:02:51 (45808): No heartbeat from core client for 30 sec - exiting 13:02:53 (45808): No heartbeat from core client for 30 sec - exiting 13:02:54 (45808): No heartbeat from core client for 30 sec - exiting 13:32:50 (43508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:32:52 (43508): No heartbeat from core client for 30 sec - exiting 13:32:53 (43508): No heartbeat from core client for 30 sec - exiting 13:42:51 (45268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:42:52 (45268): No heartbeat from core client for 30 sec - exiting 13:42:53 (45268): No heartbeat from core client for 30 sec - exiting 13:52:51 (45828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:52:52 (45828): No heartbeat from core client for 30 sec - exiting 13:57:52 (45712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:57:54 (45712): No heartbeat from core client for 30 sec - exiting 13:57:55 (45712): No heartbeat from core client for 30 sec - exiting 13:57:56 (45712): No heartbeat from core client for 30 sec - exiting 13:57:57 (45712): No heartbeat from core client for 30 sec - exiting 14:07:53 (42828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:07:55 (42828): No heartbeat from core client for 30 sec - exiting 14:07:56 (42828): No heartbeat from core client for 30 sec - exiting 14:07:57 (42828): No heartbeat from core client for 30 sec - exiting 14:07:58 (42828): No heartbeat from core client for 30 sec - exiting 14:08:00 (42828): No heartbeat from core client for 30 sec - exiting 14:12:54 (45508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:12:55 (45508): No heartbeat from core client for 30 sec - exiting 14:12:56 (45508): No heartbeat from core client for 30 sec - exiting 14:12:57 (45508): No heartbeat from core client for 30 sec - exiting 14:12:59 (45508): No heartbeat from core client for 30 sec - exiting 14:42:57 (46444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:42:58 (46444): No heartbeat from core client for 30 sec - exiting 14:43:00 (46444): No heartbeat from core client for 30 sec - exiting 14:57:58 (46240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:57:59 (46240): No heartbeat from core client for 30 sec - exiting 14:58:00 (46240): No heartbeat from core client for 30 sec - exiting 14:58:01 (46240): No heartbeat from core client for 30 sec - exiting 14:58:03 (46240): No heartbeat from core client for 30 sec - exiting 14:58:04 (46240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 14:54:23 (53460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:54:25 (53460): No heartbeat from core client for 30 sec - exiting 14:54:26 (53460): No heartbeat from core client for 30 sec - exiting 14:54:27 (53460): No heartbeat from core client for 30 sec - exiting 15:55:07 (55040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:04:46 (54120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:04:47 (54120): No heartbeat from core client for 30 sec - exiting 16:04:48 (54120): No heartbeat from core client for 30 sec - exiting 16:35:55 (54816): No heartbeat from core client for 30 sec - exiting 16:35:56 (54816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:35:58 (54816): No heartbeat from core client for 30 sec - exiting 16:37:04 (55564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:10:57 (55832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:11:00 (55832): No heartbeat from core client for 30 sec - exiting 17:11:01 (55832): No heartbeat from core client for 30 sec - exiting 17:11:02 (55832): No heartbeat from core client for 30 sec - exiting 17:11:03 (55832): No heartbeat from core client for 30 sec - exiting 17:11:05 (55832): No heartbeat from core client for 30 sec - exiting 17:11:06 (55832): No heartbeat from core client for 30 sec - exiting 17:11:07 (55832): No heartbeat from core client for 30 sec - exiting 17:51:02 (55680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:51:04 (55680): No heartbeat from core client for 30 sec - exiting 17:51:05 (55680): No heartbeat from core client for 30 sec - exiting 17:51:06 (55680): No heartbeat from core client for 30 sec - exiting 17:51:07 (55680): No heartbeat from core client for 30 sec - exiting 17:52:25 (54364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:01:02 (52832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:01:04 (52832): No heartbeat from core client for 30 sec - exiting 18:01:05 (52832): No heartbeat from core client for 30 sec - exiting 18:01:06 (52832): No heartbeat from core client for 30 sec - exiting 18:21:04 (56156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:21:07 (56156): No heartbeat from core client for 30 sec - exiting 18:21:08 (56156): No heartbeat from core client for 30 sec - exiting 18:36:13 (53524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:36:15 (53524): No heartbeat from core client for 30 sec - exiting 18:36:16 (53524): No heartbeat from core client for 30 sec - exiting 18:36:17 (53524): No heartbeat from core client for 30 sec - exiting 18:36:18 (53524): No heartbeat from core client for 30 sec - exiting 18:36:19 (53524): No heartbeat from core client for 30 sec - exiting 18:36:20 (53524): No heartbeat from core client for 30 sec - exiting 18:51:59 (55648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:52:01 (55648): No heartbeat from core client for 30 sec - exiting 18:52:02 (55648): No heartbeat from core client for 30 sec - exiting 18:52:03 (55648): No heartbeat from core client for 30 sec - exiting 18:52:05 (55648): No heartbeat from core client for 30 sec - exiting 18:52:06 (55648): No heartbeat from core client for 30 sec - exiting 18:52:07 (55648): No heartbeat from core client for 30 sec - exiting 18:53:57 (55672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:26:18 (55716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:26:19 (55716): No heartbeat from core client for 30 sec - exiting 19:26:20 (55716): No heartbeat from core client for 30 sec - exiting 19:26:22 (55716): No heartbeat from core client for 30 sec - exiting 19:36:19 (53680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:36:22 (53680): No heartbeat from core client for 30 sec - exiting 19:36:23 (53680): No heartbeat from core client for 30 sec - exiting 19:36:24 (53680): No heartbeat from core client for 30 sec - exiting 19:36:25 (53680): No heartbeat from core client for 30 sec - exiting 19:36:26 (53680): No heartbeat from core client for 30 sec - exiting 19:36:27 (53680): No heartbeat from core client for 30 sec - exiting 19:38:27 (55760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:51:21 (56044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:51:24 (56044): No heartbeat from core client for 30 sec - exiting 19:51:25 (56044): No heartbeat from core client for 30 sec - exiting 19:51:26 (56044): No heartbeat from core client for 30 sec - exiting 19:51:27 (56044): No heartbeat from core client for 30 sec - exiting 19:51:28 (56044): No heartbeat from core client for 30 sec - exiting 19:51:29 (56044): No heartbeat from core client for 30 sec - exiting 19:52:59 (55820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:56:24 (53448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:56:26 (53448): No heartbeat from core client for 30 sec - exiting 19:56:27 (53448): No heartbeat from core client for 30 sec - exiting 19:56:28 (53448): No heartbeat from core client for 30 sec - exiting 19:56:29 (53448): No heartbeat from core client for 30 sec - exiting 20:11:26 (55916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:11:28 (55916): No heartbeat from core client for 30 sec - exiting 20:11:29 (55916): No heartbeat from core client for 30 sec - exiting 20:11:30 (55916): No heartbeat from core client for 30 sec - exiting 20:11:31 (55916): No heartbeat from core client for 30 sec - exiting 20:11:33 (55916): No heartbeat from core client for 30 sec - exiting 20:11:34 (55916): No heartbeat from core client for 30 sec - exiting 20:17:09 (54824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:17:10 (54824): No heartbeat from core client for 30 sec - exiting 20:41:38 (55432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:39 (55432): No heartbeat from core client for 30 sec - exiting 20:41:40 (55432): No heartbeat from core client for 30 sec - exiting 20:41:41 (55432): No heartbeat from core client for 30 sec - exiting 20:41:43 (55432): No heartbeat from core client for 30 sec - exiting 20:43:14 (57284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:06:41 (56448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:06:44 (56448): No heartbeat from core client for 30 sec - exiting 21:06:45 (56448): No heartbeat from core client for 30 sec - exiting 21:06:46 (56448): No heartbeat from core client for 30 sec - exiting 21:06:47 (56448): No heartbeat from core client for 30 sec - exiting 21:06:48 (56448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 22:10:04 (55620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:12:20 (57252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:47 (56392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:48 (56392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 05:47:48 (54140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:47:49 (54140): No heartbeat from core client for 30 sec - exiting 05:47:50 (54140): No heartbeat from core client for 30 sec - exiting 06:25:59 (57244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:26:02 (57244): No heartbeat from core client for 30 sec - exiting 06:26:03 (57244): No heartbeat from core client for 30 sec - exiting 06:26:04 (57244): No heartbeat from core client for 30 sec - exiting 06:26:05 (57244): No heartbeat from core client for 30 sec - exiting 06:49:57 (57588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:49:59 (57588): No heartbeat from core client for 30 sec - exiting 06:50:00 (57588): No heartbeat from core client for 30 sec - exiting 06:50:01 (57588): No heartbeat from core client for 30 sec - exiting 07:26:10 (58600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:26:11 (58600): No heartbeat from core client for 30 sec - exiting 07:26:12 (58600): No heartbeat from core client for 30 sec - exiting 07:26:14 (58600): No heartbeat from core client for 30 sec - exiting 07:41:12 (58616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:41:14 (58616): No heartbeat from core client for 30 sec - exiting 07:41:15 (58616): No heartbeat from core client for 30 sec - exiting 07:51:11 (58996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:51:13 (58996): No heartbeat from core client for 30 sec - exiting 07:51:14 (58996): No heartbeat from core client for 30 sec - exiting 08:11:14 (58896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:11:17 (58896): No heartbeat from core client for 30 sec - exiting 08:11:18 (58896): No heartbeat from core client for 30 sec - exiting 08:11:19 (58896): No heartbeat from core client for 30 sec - exiting 08:27:40 (4584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4416, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:34:14 (3996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5064, iMonCtr=1 Model crash detected, will try to restart... 08:32:47 (3284): No heartbeat from core client for 30 sec - exiting 08:32:49 (3284): No heartbeat from core client for 30 sec - exiting 08:32:50 (3284): No heartbeat from core client for 30 sec - exiting 08:32:51 (3284): No heartbeat from core client for 30 sec - exiting 08:32:52 (3284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:32:53 (3284): No heartbeat from core client for 30 sec - exiting 08:32:54 (3284): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4524, iMonCtr=1 Model crash detected, will try to restart... 09:24:58 (3100): No heartbeat from core client for 30 sec - exiting 09:24:59 (3100): No heartbeat from core client for 30 sec - exiting 09:25:00 (3100): No heartbeat from core client for 30 sec - exiting 09:25:01 (3100): No heartbeat from core client for 30 sec - exiting 09:25:02 (3100): No heartbeat from core client for 30 sec - exiting 09:25:03 (3100): No heartbeat from core client for 30 sec - exiting 09:25:04 (3100): No heartbeat from core client for 30 sec - exiting 09:25:05 (3100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4624, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2604, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=364, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77D53242 read attempt to address 0x403884A8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o1hp_1940_40_007543576/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Jan 2012 08:17:32	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	777,600	1,502,834	1.9327
26 Jan 2012 06:14:27	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	751,680	1,453,268	1.9334
24 Jan 2012 07:16:57	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	725,760	1,403,982	1.9345
19 Jan 2012 03:45:30	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	699,840	1,352,368	1.9324
17 Jan 2012 07:47:57	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	673,920	1,302,305	1.9324
13 Jan 2012 04:41:49	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	648,000	1,251,247	1.9309
12 Jan 2012 02:36:33	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	622,080	1,201,374	1.9312
09 Jan 2012 23:36:13	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	596,160	1,150,768	1.9303
01 Jan 2012 13:58:32	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	570,240	1,105,811	1.9392
01 Jan 2012 00:10:28	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	544,320	1,057,894	1.9435
31 Dec 2011 10:41:54	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	518,400	1,010,032	1.9484
30 Dec 2011 15:48:06	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	492,480	961,373	1.9521
30 Dec 2011 00:02:20	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	466,560	911,678	1.9540
28 Dec 2011 03:33:42	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	440,640	861,926	1.9561
26 Dec 2011 06:24:42	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	414,720	812,067	1.9581
22 Dec 2011 22:18:02	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	388,800	761,956	1.9598
22 Dec 2011 07:58:15	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	362,880	712,347	1.9630
20 Dec 2011 08:17:43	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	336,960	662,354	1.9657
19 Dec 2011 02:01:09	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	311,040	610,351	1.9623
15 Dec 2011 04:23:11	1160027	13627204	hadcm3n_o1hp_1940_40_007543576_1	285,120	559,233	1.9614