Task 16959512

Name	hadam3p_anz_rvgh_2012_1_008967348_0
Workunit	9111523
Created	27 Aug 2014, 15:00:38 UTC
Sent	29 Aug 2014, 9:55:45 UTC
Report deadline	11 Aug 2015, 15:15:45 UTC
Received	5 Sep 2014, 4:56:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1335584
Run time	3 days 11 hours 57 min 13 sec
CPU time	3 days 0 hours 24 min 37 sec
Validate state	Invalid
Credit	1,503.36
Device peak FLOPS	2.38 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9448, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9484, selfPID=9368, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:07:07 (7172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5892, selfPID=7420, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4444, selfPID=1584, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 05:05:14 (656): No heartbeat from core client for 30 sec - exiting 05:05:15 (656): No heartbeat from core client for 30 sec - exiting 05:05:16 (656): No heartbeat from core client for 30 sec - exiting 05:05:17 (656): No heartbeat from core client for 30 sec - exiting 05:05:18 (656): No heartbeat from core client for 30 sec - exiting 05:05:19 (656): No heartbeat from core client for 30 sec - exiting 05:05:20 (656): No heartbeat from core client for 30 sec - exiting 05:05:21 (656): No heartbeat from core client for 30 sec - exiting 05:05:22 (656): No heartbeat from core client for 30 sec - exiting 05:05:23 (656): No heartbeat from core client for 30 sec - exiting 05:05:24 (656): No heartbeat from core client for 30 sec - exiting 05:05:25 (656): No heartbeat from core client for 30 sec - exiting 05:05:26 (656): No heartbeat from core client for 30 sec - exiting 05:05:27 (656): No heartbeat from core client for 30 sec - exiting 05:05:28 (656): No heartbeat from core client for 30 sec - exiting 05:05:29 (656): No heartbeat from core client for 30 sec - exiting 05:05:30 (656): No heartbeat from core client for 30 sec - exiting 05:05:31 (656): No heartbeat from core client for 30 sec - exiting 05:05:32 (656): No heartbeat from core client for 30 sec - exiting 05:05:33 (656): No heartbeat from core client for 30 sec - exiting 05:05:34 (656): No heartbeat from core client for 30 sec - exiting 05:05:35 (656): No heartbeat from core client for 30 sec - exiting 05:05:36 (656): No heartbeat from core client for 30 sec - exiting 05:05:37 (656): No heartbeat from core client for 30 sec - exiting 05:05:38 (656): No heartbeat from core client for 30 sec - exiting 05:05:39 (656): No heartbeat from core client for 30 sec - exiting 05:05:40 (656): No heartbeat from core client for 30 sec - exiting 05:05:41 (656): No heartbeat from core client for 30 sec - exiting 05:05:42 (656): No heartbeat from core client for 30 sec - exiting 05:05:43 (656): No heartbeat from core client for 30 sec - exiting 05:05:44 (656): No heartbeat from core client for 30 sec - exiting 05:05:45 (656): No heartbeat from core client for 30 sec - exiting 05:05:46 (656): No heartbeat from core client for 30 sec - exiting 05:05:47 (656): No heartbeat from core client for 30 sec - exiting 05:05:48 (656): No heartbeat from core client for 30 sec - exiting 05:05:49 (656): No heartbeat from core client for 30 sec - exiting 05:05:50 (656): No heartbeat from core client for 30 sec - exiting 05:05:51 (656): No heartbeat from core client for 30 sec - exiting 05:05:52 (656): No heartbeat from core client for 30 sec - exiting 05:05:53 (656): No heartbeat from core client for 30 sec - exiting 05:05:54 (656): No heartbeat from core client for 30 sec - exiting 05:05:55 (656): No heartbeat from core client for 30 sec - exiting 05:05:56 (656): No heartbeat from core client for 30 sec - exiting 05:05:57 (656): No heartbeat from core client for 30 sec - exiting 05:05:58 (656): No heartbeat from core client for 30 sec - exiting 05:05:59 (656): No heartbeat from core client for 30 sec - exiting 05:06:00 (656): No heartbeat from core client for 30 sec - exiting 05:06:01 (656): No heartbeat from core client for 30 sec - exiting 05:06:02 (656): No heartbeat from core client for 30 sec - exiting 05:06:03 (656): No heartbeat from core client for 30 sec - exiting 05:06:04 (656): No heartbeat from core client for 30 sec - exiting 05:06:05 (656): No heartbeat from core client for 30 sec - exiting 05:06:06 (656): No heartbeat from core client for 30 sec - exiting 05:06:07 (656): No heartbeat from core client for 30 sec - exiting 05:06:08 (656): No heartbeat from core client for 30 sec - exiting 05:06:09 (656): No heartbeat from core client for 30 sec - exiting 05:06:10 (656): No heartbeat from core client for 30 sec - exiting 05:06:11 (656): No heartbeat from core client for 30 sec - exiting 05:06:12 (656): No heartbeat from core client for 30 sec - exiting 05:06:13 (656): No heartbeat from core client for 30 sec - exiting 05:06:14 (656): No heartbeat from core client for 30 sec - exiting 05:06:15 (656): No heartbeat from core client for 30 sec - exiting 05:06:16 (656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:06:17 (656): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 05:19:43 (6568): No heartbeat from core client for 30 sec - exiting 05:19:44 (6568): No heartbeat from core client for 30 sec - exiting 05:19:45 (6568): No heartbeat from core client for 30 sec - exiting 05:19:46 (6568): No heartbeat from core client for 30 sec - exiting 05:19:48 (6568): No heartbeat from core client for 30 sec - exiting 05:19:49 (6568): No heartbeat from core client for 30 sec - exiting 05:19:50 (6568): No heartbeat from core client for 30 sec - exiting 05:19:51 (6568): No heartbeat from core client for 30 sec - exiting 05:19:52 (6568): No heartbeat from core client for 30 sec - exiting 05:19:53 (6568): No heartbeat from core client for 30 sec - exiting 05:19:54 (6568): No heartbeat from core client for 30 sec - exiting 05:19:55 (6568): No heartbeat from core client for 30 sec - exiting 05:19:56 (6568): No heartbeat from core client for 30 sec - exiting 05:19:57 (6568): No heartbeat from core client for 30 sec - exiting 05:19:58 (6568): No heartbeat from core client for 30 sec - exiting 05:20:00 (6568): No heartbeat from core client for 30 sec - exiting 05:20:01 (6568): No heartbeat from core client for 30 sec - exiting 05:20:02 (6568): No heartbeat from core client for 30 sec - exiting 05:20:03 (6568): No heartbeat from core client for 30 sec - exiting 05:20:04 (6568): No heartbeat from core client for 30 sec - exiting 05:20:05 (6568): No heartbeat from core client for 30 sec - exiting 05:20:06 (6568): No heartbeat from core client for 30 sec - exiting 05:20:07 (6568): No heartbeat from core client for 30 sec - exiting 05:20:08 (6568): No heartbeat from core client for 30 sec - exiting 05:20:09 (6568): No heartbeat from core client for 30 sec - exiting 05:20:10 (6568): No heartbeat from core client for 30 sec - exiting 05:20:12 (6568): No heartbeat from core client for 30 sec - exiting 05:20:13 (6568): No heartbeat from core client for 30 sec - exiting 05:20:14 (6568): No heartbeat from core client for 30 sec - exiting 05:20:15 (6568): No heartbeat from core client for 30 sec - exiting 05:20:16 (6568): No heartbeat from core client for 30 sec - exiting 05:20:17 (6568): No heartbeat from core client for 30 sec - exiting 05:20:18 (6568): No heartbeat from core client for 30 sec - exiting 05:20:19 (6568): No heartbeat from core client for 30 sec - exiting 05:20:20 (6568): No heartbeat from core client for 30 sec - exiting 05:20:21 (6568): No heartbeat from core client for 30 sec - exiting 05:20:23 (6568): No heartbeat from core client for 30 sec - exiting 05:20:24 (6568): No heartbeat from core client for 30 sec - exiting 05:20:25 (6568): No heartbeat from core client for 30 sec - exiting 05:20:26 (6568): No heartbeat from core client for 30 sec - exiting 05:20:27 (6568): No heartbeat from core client for 30 sec - exiting 05:20:28 (6568): No heartbeat from core client for 30 sec - exiting 05:20:29 (6568): No heartbeat from core client for 30 sec - exiting 05:20:30 (6568): No heartbeat from core client for 30 sec - exiting 05:20:31 (6568): No heartbeat from core client for 30 sec - exiting 05:20:32 (6568): No heartbeat from core client for 30 sec - exiting 05:20:33 (6568): No heartbeat from core client for 30 sec - exiting 05:20:35 (6568): No heartbeat from core client for 30 sec - exiting 05:20:36 (6568): No heartbeat from core client for 30 sec - exiting 05:20:37 (6568): No heartbeat from core client for 30 sec - exiting 05:20:38 (6568): No heartbeat from core client for 30 sec - exiting 05:20:39 (6568): No heartbeat from core client for 30 sec - exiting 05:20:40 (6568): No heartbeat from core client for 30 sec - exiting 05:20:41 (6568): No heartbeat from core client for 30 sec - exiting 05:20:42 (6568): No heartbeat from core client for 30 sec - exiting 05:20:43 (6568): No heartbeat from core client for 30 sec - exiting 05:20:44 (6568): No heartbeat from core client for 30 sec - exiting 05:20:45 (6568): No heartbeat from core client for 30 sec - exiting 05:20:46 (6568): No heartbeat from core client for 30 sec - exiting 05:20:47 (6568): No heartbeat from core client for 30 sec - exiting 05:20:48 (6568): No heartbeat from core client for 30 sec - exiting 05:20:49 (6568): No heartbeat from core client for 30 sec - exiting 05:20:50 (6568): No heartbeat from core client for 30 sec - exiting 05:20:51 (6568): No heartbeat from core client for 30 sec - exiting 05:20:52 (6568): No heartbeat from core client for 30 sec - exiting 05:20:53 (6568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:21:49 (2100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:35:31 (5156): No heartbeat from core client for 30 sec - exiting 06:35:32 (5156): No heartbeat from core client for 30 sec - exiting 06:35:33 (5156): No heartbeat from core client for 30 sec - exiting 06:35:34 (5156): No heartbeat from core client for 30 sec - exiting 06:35:35 (5156): No heartbeat from core client for 30 sec - exiting 06:35:36 (5156): No heartbeat from core client for 30 sec - exiting 06:35:37 (5156): No heartbeat from core client for 30 sec - exiting 06:35:38 (5156): No heartbeat from core client for 30 sec - exiting 06:35:39 (5156): No heartbeat from core client for 30 sec - exiting 06:35:40 (5156): No heartbeat from core client for 30 sec - exiting 06:35:42 (5156): No heartbeat from core client for 30 sec - exiting 06:35:43 (5156): No heartbeat from core client for 30 sec - exiting 06:35:44 (5156): No heartbeat from core client for 30 sec - exiting 06:35:45 (5156): No heartbeat from core client for 30 sec - exiting 06:35:46 (5156): No heartbeat from core client for 30 sec - exiting 06:35:47 (5156): No heartbeat from core client for 30 sec - exiting 06:35:48 (5156): No heartbeat from core client for 30 sec - exiting 06:35:49 (5156): No heartbeat from core client for 30 sec - exiting 06:35:50 (5156): No heartbeat from core client for 30 sec - exiting 06:35:51 (5156): No heartbeat from core client for 30 sec - exiting 06:35:52 (5156): No heartbeat from core client for 30 sec - exiting 06:35:54 (5156): No heartbeat from core client for 30 sec - exiting 06:35:55 (5156): No heartbeat from core client for 30 sec - exiting 06:35:56 (5156): No heartbeat from core client for 30 sec - exiting 06:35:57 (5156): No heartbeat from core client for 30 sec - exiting 06:35:58 (5156): No heartbeat from core client for 30 sec - exiting 06:35:59 (5156): No heartbeat from core client for 30 sec - exiting 06:36:00 (5156): No heartbeat from core client for 30 sec - exiting 06:36:01 (5156): No heartbeat from core client for 30 sec - exiting 06:36:02 (5156): No heartbeat from core client for 30 sec - exiting 06:36:03 (5156): No heartbeat from core client for 30 sec - exiting 06:36:04 (5156): No heartbeat from core client for 30 sec - exiting 06:36:06 (5156): No heartbeat from core client for 30 sec - exiting 06:36:07 (5156): No heartbeat from core client for 30 sec - exiting 06:36:08 (5156): No heartbeat from core client for 30 sec - exiting 06:36:09 (5156): No heartbeat from core client for 30 sec - exiting 06:36:10 (5156): No heartbeat from core client for 30 sec - exiting 06:36:11 (5156): No heartbeat from core client for 30 sec - exiting 06:36:12 (5156): No heartbeat from core client for 30 sec - exiting 06:36:13 (5156): No heartbeat from core client for 30 sec - exiting 06:36:14 (5156): No heartbeat from core client for 30 sec - exiting 06:36:15 (5156): No heartbeat from core client for 30 sec - exiting 06:36:17 (5156): No heartbeat from core client for 30 sec - exiting 06:36:18 (5156): No heartbeat from core client for 30 sec - exiting 06:36:19 (5156): No heartbeat from core client for 30 sec - exiting 06:36:20 (5156): No heartbeat from core client for 30 sec - exiting 06:36:21 (5156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8180, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2740, selfPID=8100, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_rvgh_2012_1_008967348_0_4.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_rvgh_2012_1_008967348_0_5.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_rvgh_2012_1_008967348_0_6.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_rvgh_2012_1_008967348_0_7.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_rvgh_2012_1_008967348_0_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_rvgh_2012_1_008967348_0_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_rvgh_2012_1_008967348_0_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_rvgh_2012_1_008967348_0_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_rvgh_2012_1_008967348_0_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Sep 2014 17:00:05	1335584	16959512	hadam3p_anz_rvgh_2012_1_008967348_0	34,859	249,688	7.1628
01 Sep 2014 17:36:19	1335584	16959512	hadam3p_anz_rvgh_2012_1_008967348_0	23,339	166,784	7.1462
31 Aug 2014 06:39:14	1335584	16959512	hadam3p_anz_rvgh_2012_1_008967348_0	11,819	84,971	7.1894