Task 15400288

Name	hadcm3n_o0uc_2060_40_008239532_1
Workunit	8394656
Created	26 Oct 2012, 18:01:50 UTC
Sent	26 Oct 2012, 18:02:02 UTC
Report deadline	26 Jan 2013, 1:29:13 UTC
Received	10 Jan 2013, 1:14:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1179495
Run time	22 days 10 hours 47 min 9 sec
CPU time	19 days 0 hours 17 min 32 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.40 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1328, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3168, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3168, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4408, iMonCtr=1 Model crash det22:52:43 (2992): No heartbeat from core client for 30 sec - exiting 22:52:45 (2992): No heartbeat from core client for 30 sec - exiting 22:52:46 (2992): No heartbeat from core client for 30 sec - exiting 22:52:47 (2992): No heartbeat from core client for 30 sec - exiting 22:52:48 (2992): No heartbeat from core client for 30 sec - exiting 22:52:49 (2992): No heartbeat from core client for 30 sec - exiting 22:52:50 (2992): No heartbeat from core client for 30 sec - exiting 22:52:51 (2992): No heartbeat from core client for 30 sec - exiting 22:52:52 (2992): No heartbeat from core client for 30 sec - exiting 22:52:53 (2992): No heartbeat from core client for 30 sec - exiting 22:52:54 (2992): No heartbeat from core client for 30 sec - exiting 22:52:55 (2992): No heartbeat from core client for 30 sec - exiting 22:52:56 (2992): No heartbeat from core client for 30 sec - exiting 22:52:57 (2992): No heartbeat from core client for 30 sec - exiting 22:52:59 (2992): No heartbeat from core client for 30 sec - exiting 22:53:00 (2992): No heartbeat from core client for 30 sec - exiting 22:53:01 (2992): No heartbeat from core client for 30 sec - exiting 22:53:02 (2992): No heartbeat from core client for 30 sec - exiting 22:53:03 (2992): No heartbeat from core client for 30 sec - exiting 22:53:04 (2992): No heartbeat from core client for 30 sec - exiting 22:53:05 (2992): No heartbeat from core client for 30 sec - exiting 22:53:06 (2992): No heartbeat from core client for 30 sec - exiting 22:53:07 (2992): No heartbeat from core client for 30 sec - exiting 22:53:08 (2992): No heartbeat from core client for 30 sec - exiting 22:53:10 (2992): No heartbeat from core client for 30 sec - exiting 22:53:11 (2992): No heartbeat from core client for 30 sec - exiting 22:53:12 (2992): No heartbeat from core client for 30 sec - exiting 22:53:13 (2992): No heartbeat from core client for 30 sec - exiting 22:53:14 (2992): No heartbeat from core client for 30 sec - exiting 22:53:15 (2992): No heartbeat from core client for 30 sec - exiting 22:53:16 (2992): No heartbeat from core client for 30 sec - exiting 22:53:17 (2992): No heartbeat from core client for 30 sec - exiting 22:53:18 (2992): No heartbeat from core client for 30 sec - exiting 22:53:19 (2992): No heartbeat from core client for 30 sec - exiting 22:53:20 (2992): No heartbeat from core client for 30 sec - exiting 22:53:22 (2992): No heartbeat from core client for 30 sec - exiting 22:53:23 (2992): No heartbeat from core client for 30 sec - exiting 22:53:24 (2992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:46 (1272): No heartbeat from core client for 30 sec - exiting 20:16:47 (1272): No heartbeat from core client for 30 sec - exiting 20:16:48 (1272): No heartbeat from core client for 30 sec - exiting 20:16:49 (1272): No heartbeat from core client for 30 sec - exiting 20:16:50 (1272): No heartbeat from core client for 30 sec - exiting 20:16:51 (1272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:48:35 (4964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:51:03 (6824): No heartbeat from core client for 30 sec - exiting 07:51:04 (6824): No heartbeat from core client for 30 sec - exiting 07:51:06 (6824): No heartbeat from core client for 30 sec - exiting 07:51:07 (6824): No heartbeat from core client for 30 sec - exiting 07:51:08 (6824): No heartbeat from core client for 30 sec - exiting 07:51:09 (6824): No heartbeat from core client for 30 sec - exiting 07:51:10 (6824): No heartbeat from core client for 30 sec - exiting 07:51:11 (6824): No heartbeat from core client for 30 sec - exiting 07:51:12 (6824): No heartbeat from core client for 30 sec - exiting 07:51:13 (6824): No heartbeat from core client for 30 sec - exiting 07:51:14 (6824): No heartbeat from core client for 30 sec - exiting 07:51:15 (6824): No heartbeat from core client for 30 sec - exiting 07:51:17 (6824): No heartbeat from core client for 30 sec - exiting 07:51:18 (6824): No heartbeat from core client for 30 sec - exiting 07:51:19 (6824): No heartbeat from core client for 30 sec - exiting 07:51:20 (6824): No heartbeat from core client for 30 sec - exiting 07:51:21 (6824): No heartbeat from core client for 30 sec - exiting 07:51:22 (6824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:41:02 (264): No heartbeat from core client for 30 sec - exiting 20:41:05 (264): No heartbeat from core client for 30 sec - exiting 20:41:06 (264): No heartbeat from core client for 30 sec - exiting 20:41:07 (264): No heartbeat from core client for 30 sec - exiting 20:41:08 (264): No heartbeat from core client for 30 sec - exiting 20:41:09 (264): No heartbeat from core client for 30 sec - exiting 20:41:10 (264): No heartbeat from core client for 30 sec - exiting 20:41:11 (264): No heartbeat from core client for 30 sec - exiting 20:41:12 (264): No heartbeat from core client for 30 sec - exiting 20:41:13 (264): No heartbeat from core client for 30 sec - exiting 20:41:14 (264): No heartbeat from core client for 30 sec - exiting 20:41:15 (264): No heartbeat from core client for 30 sec - exiting 20:41:17 (264): No heartbeat from core client for 30 sec - exiting 20:41:18 (264): No heartbeat from core client for 30 sec - exiting 20:41:19 (264): No heartbeat from core client for 30 sec - exiting 20:41:20 (264): No heartbeat from core client for 30 sec - exiting 20:41:21 (264): No heartbeat from core client for 30 sec - exiting 20:41:22 (264): No heartbeat from core client for 30 sec - exiting 20:41:23 (264): No heartbeat from core client for 30 sec - exiting 20:41:24 (264): No heartbeat from core client for 30 sec - exiting 20:41:25 (264): No heartbeat from core client for 30 sec - exiting 20:41:26 (264): No heartbeat from core client for 30 sec - exiting 20:41:28 (264): No heartbeat from core client for 30 sec - exiting 20:41:29 (264): No heartbeat from core client for 30 sec - exiting 20:41:30 (264): No heartbeat from core client for 30 sec - exiting 20:41:31 (264): No heartbeat from core client for 30 sec - exiting 20:41:32 (264): No heartbeat from core client for 30 sec - exiting 20:41:33 (264): No heartbeat from core client for 30 sec - exiting 20:41:34 (264): No heartbeat from core client for 30 sec - exiting 20:41:35 (264): No heartbeat from core client for 30 sec - exiting 20:41:36 (264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:00:04 (3216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:40:31 (2960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:15:18 (8196): No heartbeat from core client for 30 sec - exiting 19:15:19 (8196): No heartbeat from core client for 30 sec - exiting 19:15:20 (8196): No heartbeat from core client for 30 sec - exiting 19:15:21 (8196): No heartbeat from core client for 30 sec - exiting 19:15:23 (8196): No heartbeat from core client for 30 sec - exiting 19:15:24 (8196): No heartbeat from core client for 30 sec - exiting 19:15:25 (8196): No heartbeat from core client for 30 sec - exiting 19:15:26 (8196): No heartbeat from core client for 30 sec - exiting 19:15:27 (8196): No heartbeat from core client for 30 sec - exiting 19:15:28 (8196): No heartbeat from core client for 30 sec - exiting 19:15:29 (8196): No heartbeat from core client for 30 sec - exiting 19:15:30 (8196): No heartbeat from core client for 30 sec - exiting 19:15:31 (8196): No heartbeat from core client for 30 sec - exiting 19:15:32 (8196): No heartbeat from core client for 30 sec - exiting 19:15:33 (8196): No heartbeat from core client for 30 sec - exiting 19:15:35 (8196): No heartbeat from core client for 30 sec - exiting 19:15:36 (8196): No heartbeat from core client for 30 sec - exiting 19:15:37 (8196): No heartbeat from core client for 30 sec - exiting 19:15:38 (8196): No heartbeat from core client for 30 sec - exiting 19:15:39 (8196): No heartbeat from core client for 30 sec - exiting 19:15:40 (8196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:15:41 (8196): No heartbeat from core client for 30 sec - exiting 19:15:42 (8196): No heartbeat from core client for 30 sec - exiting 19:15:43 (8196): No heartbeat from core client for 30 sec - exiting 19:15:44 (8196): No heartbeat from core client for 30 sec - exiting 19:15:45 (8196): No heartbeat from core client for 30 sec - exiting 19:15:47 (8196): No heartbeat from core client for 30 sec - exiting 19:15:48 (8196): No heartbeat from core client for 30 sec - exiting 19:15:49 (8196): No heartbeat from core client for 30 sec - exiting 19:15:50 (8196): No heartbeat from core client for 30 sec - exiting 19:15:51 (8196): No heartbeat from core client for 30 sec - exiting 19:17:15 (1388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3164, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:09:16 (3132): No heartbeat from core client for 30 sec - exiting 04:09:18 (3132): No heartbeat from core client for 30 sec - exiting 04:09:19 (3132): No heartbeat from core client for 30 sec - exiting 04:09:20 (3132): No heartbeat from core client for 30 sec - exiting 04:09:21 (3132): No heartbeat from core client for 30 sec - exiting 04:09:22 (3132): No heartbeat from core client for 30 sec - exiting 04:09:23 (3132): No heartbeat from core client for 30 sec - exiting 04:09:24 (3132): No heartbeat from core client for 30 sec - exiting 04:09:25 (3132): No heartbeat from core client for 30 sec - exiting 04:09:26 (3132): No heartbeat from core client for 30 sec - exiting 04:09:27 (3132): No heartbeat from core client for 30 sec - exiting 04:09:28 (3132): No heartbeat from core client for 30 sec - exiting 04:09:29 (3132): No heartbeat from core client for 30 sec - exiting 04:09:30 (3132): No heartbeat from core client for 30 sec - exiting 04:09:31 (3132): No heartbeat from core client for 30 sec - exiting 04:09:32 (3132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2160, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1248, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3208, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4032, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2516, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3108, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1600, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o0ucko.pjs8c10 Error converting file to netcdf: dataout/o0ucko.pis8c10 Error converting file to netcdf: dataout/o0ucko.pfs8c10 Error converting file to netcdf: dataout/o0ucka.phs8c10 Error converting file to netcdf: dataout/o0ucka.pgs8c10 Error converting file to netcdf: dataout/o0ucka.pes8c10 Error converting file to netcdf: dataout/o0ucka.pds8c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Jan 2013 00:17:09	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	777,600	1,642,636	2.1124
07 Jan 2013 16:52:06	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	751,680	1,581,713	2.1042
05 Jan 2013 21:37:34	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	725,760	1,523,776	2.0996
04 Jan 2013 15:56:20	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	699,840	1,463,439	2.0911
01 Jan 2013 20:25:19	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	673,920	1,407,529	2.0886
31 Dec 2012 19:14:38	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	648,000	1,348,324	2.0807
29 Dec 2012 19:38:06	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	622,080	1,306,438	2.1001
27 Dec 2012 20:24:35	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	596,160	1,250,244	2.0972
25 Dec 2012 21:56:13	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	570,240	1,190,141	2.0871
24 Dec 2012 18:21:59	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	544,320	1,131,488	2.0787
24 Dec 2012 00:44:22	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	518,400	1,074,767	2.0732
23 Dec 2012 06:32:34	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	492,480	1,014,763	2.0605
21 Dec 2012 23:42:16	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	466,560	958,640	2.0547
19 Dec 2012 01:35:39	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	440,640	901,851	2.0467
16 Dec 2012 18:17:34	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	414,720	847,847	2.0444
14 Dec 2012 20:24:49	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	388,800	799,883	2.0573
14 Dec 2012 20:24:49	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	362,880	751,945	2.0722
14 Dec 2012 20:24:49	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	336,960	697,568	2.0702
06 Dec 2012 18:21:35	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	311,040	642,026	2.0641
01 Dec 2012 19:01:21	1179495	15400288	hadcm3n_o0uc_2060_40_008239532_1	285,120	587,744	2.0614