Task 13496017

Name	hadam3p_eu_6rw8_2002_1_007503421_0
Workunit	7700896
Created	14 Oct 2011, 20:40:35 UTC
Sent	15 Oct 2011, 12:14:02 UTC
Report deadline	26 Sep 2012, 17:34:02 UTC
Received	25 Oct 2011, 14:49:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1103559
Run time	3 days 3 hours 17 min 56 sec
CPU time	2 days 9 hours 25 min 33 sec
Validate state	Invalid
Credit	1,194.02
Device peak FLOPS	1.98 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Europe v6.09 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7852, iMonCtr=2 Model crash detected, will try to restart... 20:09:00 (6292): No heartbeat from core client for 30 sec - exiting 20:09:01 (6292): No heartbeat from core client for 30 sec - exiting 20:09:02 (6292): No heartbeat from core client for 30 sec - exiting 20:09:03 (6292): No heartbeat from core client for 30 sec - exiting 20:09:04 (6292): No heartbeat from core client for 30 sec - exiting 20:09:05 (6292): No heartbeat from core client for 30 sec - exiting 20:09:06 (6292): No heartbeat from core client for 30 sec - exiting 20:09:07 (6292): No heartbeat from core client for 30 sec - exiting 20:09:08 (6292): No heartbeat from core client for 30 sec - exiting 20:09:09 (6292): No heartbeat from core client for 30 sec - exiting 20:09:10 (6292): No heartbeat from core client for 30 sec - exiting 20:09:11 (6292): No heartbeat from core client for 30 sec - exiting 20:09:12 (6292): No heartbeat from core client for 30 sec - exiting 20:09:13 (6292): No heartbeat from core client for 30 sec - exiting 20:09:14 (6292): No heartbeat from core client for 30 sec - exiting 20:09:15 (6292): No heartbeat from core client for 30 sec - exiting 20:09:16 (6292): No heartbeat from core client for 30 sec - exiting 20:09:17 (6292): No heartbeat from core client for 30 sec - exiting 20:09:18 (6292): No heartbeat from core client for 30 sec - exiting 20:09:19 (6292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:20:36 (3472): No heartbeat from core client for 30 sec - exiting 21:20:37 (3472): No heartbeat from core client for 30 sec - exiting 21:20:38 (3472): No heartbeat from core client for 30 sec - exiting 21:20:39 (3472): No heartbeat from core client for 30 sec - exiting 21:20:40 (3472): No heartbeat from core client for 30 sec - exiting 21:20:41 (3472): No heartbeat from core client for 30 sec - exiting 21:20:42 (3472): No heartbeat from core client for 30 sec - exiting 21:20:43 (3472): No heartbeat from core client for 30 sec - exiting 21:20:44 (3472): No heartbeat from core client for 30 sec - exiting 21:20:45 (3472): No heartbeat from core client for 30 sec - exiting 21:20:46 (3472): No heartbeat from core client for 30 sec - exiting 21:20:47 (3472): No heartbeat from core client for 30 sec - exiting 21:20:48 (3472): No heartbeat from core client for 30 sec - exiting 21:20:49 (3472): No heartbeat from core client for 30 sec - exiting 21:20:50 (3472): No heartbeat from core client for 30 sec - exiting 21:20:51 (3472): No heartbeat from core client for 30 sec - exiting 21:20:52 (3472): No heartbeat from core client for 30 sec - exiting 21:20:53 (3472): No heartbeat from core client for 30 sec - exiting 21:20:54 (3472): No heartbeat from core client for 30 sec - exiting 21:20:55 (3472): No heartbeat from core client for 30 sec - exiting 21:20:56 (3472): No heartbeat from core client for 30 sec - exiting 21:20:57 (3472): No heartbeat from core client for 30 sec - exiting 21:20:58 (3472): No heartbeat from core client for 30 sec - exiting 21:20:59 (3472): No heartbeat from core client for 30 sec - exiting 21:21:00 (3472): No heartbeat from core client for 30 sec - exiting 21:21:01 (3472): No heartbeat from core client for 30 sec - exiting 21:21:02 (3472): No heartbeat from core client for 30 sec - exiting 21:21:03 (3472): No heartbeat from core client for 30 sec - exiting 21:21:04 (3472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3608, selfPID=4532, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 11:44:54 (5420): No heartbeat from core client for 30 sec - exiting 11:44:56 (5420): No heartbeat from core client for 30 sec - exiting 11:44:57 (5420): No heartbeat from core client for 30 sec - exiting 11:44:58 (5420): No heartbeat from core client for 30 sec - exiting 11:44:59 (5420): No heartbeat from core client for 30 sec - exiting 11:45:00 (5420): No heartbeat from core client for 30 sec - exiting 11:45:01 (5420): No heartbeat from core client for 30 sec - exiting 11:45:02 (5420): No heartbeat from core client for 30 sec - exiting 11:45:03 (5420): No heartbeat from core client for 30 sec - exiting 11:45:04 (5420): No heartbeat from core client for 30 sec - exiting 11:45:05 (5420): No heartbeat from core client for 30 sec - exiting 11:45:06 (5420): No heartbeat from core client for 30 sec - exiting 11:45:07 (5420): No heartbeat from core client for 30 sec - exiting 11:45:08 (5420): No heartbeat from core client for 30 sec - exiting 11:45:09 (5420): No heartbeat from core client for 30 sec - exiting 11:45:10 (5420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:06:27 (4196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6372, selfPID=6372, iMonCtr=2 00:06:28 (4196): No heartbeat from core client for 30 sec - exiting 00:24:16 (4952): No heartbeat from core client for 30 sec - exiting 00:24:18 (4952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:04:27 (5332): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:40:17 (7328): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... GController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8412, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:22:38 (7484): No heartbeat from core client for 30 sec - exiting 20:22:39 (7484): No heartbeat from core client for 30 sec - exiting 20:22:40 (7484): No heartbeat from core client for 30 sec - exiting 20:22:41 (7484): No heartbeat from core client for 30 sec - exiting 20:22:42 (7484): No heartbeat from core client for 30 sec - exiting 20:22:43 (7484): No heartbeat from core client for 30 sec - exiting 20:22:44 (7484): No heartbeat from core client for 30 sec - exiting 20:22:45 (7484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 09:54:49 (4984): No heartbeat from core client for 30 sec - exiting 09:54:50 (4984): No heartbeat from core client for 30 sec - exiting 09:54:51 (4984): No heartbeat from core client for 30 sec - exiting 09:54:52 (4984): No heartbeat from core client for 30 sec - exiting 09:54:53 (4984): No heartbeat from core client for 30 sec - exiting 09:54:54 (4984): No heartbeat from core client for 30 sec - exiting 09:54:55 (4984): No heartbeat from core client for 30 sec - exiting 09:54:56 (4984): No heartbeat from core client for 30 sec - exiting 09:54:57 (4984): No heartbeat from core client for 30 sec - exiting 09:54:58 (4984): No heartbeat from core client for 30 sec - exiting 09:54:59 (4984): No heartbeat from core client for 30 sec - exiting 09:55:00 (4984): No heartbeat from core client for 30 sec - exiting 09:55:01 (4984): No heartbeat from core client for 30 sec - exiting 09:55:02 (4984): No heartbeat from core client for 30 sec - exiting 09:55:03 (4984): No heartbeat from core client for 30 sec - exiting 09:55:04 (4984): No heartbeat from core client for 30 sec - exiting 09:55:05 (4984): No heartbeat from core client for 30 sec - exiting 09:55:06 (4984): No heartbeat from core client for 30 sec - exiting 09:55:07 (4984): No heartbeat from core client for 30 sec - exiting 09:55:08 (4984): No heartbeat from core client for 30 sec - exiting 09:55:09 (4984): No heartbeat from core client for 30 sec - exiting 09:55:10 (4984): No heartbeat from core client for 30 sec - exiting 09:55:11 (4984): No heartbeat from core client for 30 sec - exiting 09:55:12 (4984): No heartbeat from core client for 30 sec - exiting 09:55:13 (4984): No heartbeat from core client for 30 sec - exiting 09:55:14 (4984): No heartbeat from core client for 30 sec - exiting 09:55:15 (4984): No heartbeat from core client for 30 sec - exiting 09:55:16 (4984): No heartbeat from core client for 30 sec - exiting 09:55:17 (4984): No heartbeat from core client for 30 sec - exiting 09:55:18 (4984): No heartbeat from core client for 30 sec - exiting 09:56:45 (4984): No heartbeat from core client for 30 sec - exiting 09:56:46 (4984): No heartbeat from core client for 30 sec - exiting 09:56:47 (4984): No heartbeat from core client for 30 sec - exiting 09:56:48 (4984): No heartbeat from core client for 30 sec - exiting 09:56:49 (4984): No heartbeat from core client for 30 sec - exiting 09:56:50 (4984): No heartbeat from core client for 30 sec - exiting 09:56:51 (4984): No heartbeat from core client for 30 sec - exiting 09:56:52 (4984): No heartbeat from core client for 30 sec - exiting 09:56:53 (4984): No heartbeat from core client for 30 sec - exiting 09:56:54 (4984): No heartbeat from core client for 30 sec - exiting 09:56:55 (4984): No heartbeat from core client for 30 sec - exiting 09:56:56 (4984): No heartbeat from core client for 30 sec - exiting 09:56:57 (4984): No heartbeat from core client for 30 sec - exiting 09:56:58 (4984): No heartbeat from core client for 30 sec - exiting 09:56:59 (4984): No heartbeat from core client for 30 sec - exiting 09:57:00 (4984): No heartbeat from core client for 30 sec - exiting 09:57:01 (4984): No heartbeat from core client for 30 sec - exiting 09:57:02 (4984): No heartbeat from core client for 30 sec - exiting 09:57:03 (4984): No heartbeat from core client for 30 sec - exiting 09:57:04 (4984): No heartbeat from core client for 30 sec - exiting 09:57:05 (4984): No heartbeat from core client for 30 sec - exiting 09:57:06 (4984): No heartbeat from core client for 30 sec - exiting 09:57:07 (4984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8160, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6572, selfPID=5288, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_eu_6rw8_2002_1_007503421_0_7.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_6rw8_2002_1_007503421_0_8.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_6rw8_2002_1_007503421_0_9.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_6rw8_2002_1_007503421_0_10.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_6rw8_2002_1_007503421_0_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_6rw8_2002_1_007503421_0_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Oct 2011 13:05:11	1103559	13496017	hadam3p_eu_6rw8_2002_1_007503421_0	69,216	205,026	2.9621
31 Oct 2011 13:05:10	1103559	13496017	hadam3p_eu_6rw8_2002_1_007503421_0	57,696	170,919	2.9624
31 Oct 2011 13:05:10	1103559	13496017	hadam3p_eu_6rw8_2002_1_007503421_0	46,176	136,781	2.9622
31 Oct 2011 13:05:10	1103559	13496017	hadam3p_eu_6rw8_2002_1_007503421_0	34,656	102,580	2.9599
31 Oct 2011 13:05:10	1103559	13496017	hadam3p_eu_6rw8_2002_1_007503421_0	23,136	68,550	2.9629
18 Oct 2011 21:06:38	1103559	13496017	hadam3p_eu_6rw8_2002_1_007503421_0	11,616	35,009	3.0139