Task 16416503

Name	hadam3p_anz_n8y3_2012_1_008599211_0
Workunit	8745723
Created	26 Mar 2014, 19:04:30 UTC
Sent	27 Mar 2014, 17:34:53 UTC
Report deadline	9 Mar 2015, 22:54:53 UTC
Received	7 Apr 2014, 1:14:17 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1315213
Run time	4 days 2 hours 52 min 58 sec
CPU time	3 days 18 hours 58 min 36 sec
Validate state	Invalid
Credit	1,006.54
Device peak FLOPS	1.80 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9336, selfPID=8716, iMonCtr=1 Model crash detected, will try to restart... 11:19:04 (3412): No heartbeat from core client for 30 sec - exiting 11:19:05 (3412): No heartbeat from core client for 30 sec - exiting 11:19:06 (3412): No heartbeat from core client for 30 sec - exiting 11:19:07 (3412): No heartbeat from core client for 30 sec - exiting 11:19:08 (3412): No heartbeat from core client for 30 sec - exiting 11:19:09 (3412): No heartbeat from core client for 30 sec - exiting 11:19:10 (3412): No heartbeat from core client for 30 sec - exiting 11:19:58 (3412): No heartbeat from core client for 30 sec - exiting 11:19:59 (3412): No heartbeat from core client for 30 sec - exiting 11:20:00 (3412): No heartbeat from core client for 30 sec - exiting 11:20:01 (3412): No heartbeat from core client for 30 sec - exiting 11:20:02 (3412): No heartbeat from core client for 30 sec - exiting 11:20:03 (3412): No heartbeat from core client for 30 sec - exiting 11:20:04 (3412): No heartbeat from core client for 30 sec - exiting 11:20:05 (3412): No heartbeat from core client for 30 sec - exiting 11:20:06 (3412): No heartbeat from core client for 30 sec - exiting 11:20:07 (3412): No heartbeat from core client for 30 sec - exiting 11:20:08 (3412): No heartbeat from core client for 30 sec - exiting 11:20:09 (3412): No heartbeat from core client for 30 sec - exiting 11:20:10 (3412): No heartbeat from core client for 30 sec - exiting 11:20:11 (3412): No heartbeat from core client for 30 sec - exiting 11:20:12 (3412): No heartbeat from core client for 30 sec - exiting 11:20:13 (3412): No heartbeat from core client for 30 sec - exiting 11:20:14 (3412): No heartbeat from core client for 30 sec - exiting 11:20:15 (3412): No heartbeat from core client for 30 sec - exiting 11:20:16 (3412): No heartbeat from core client for 30 sec - exiting 11:20:17 (3412): No heartbeat from core client for 30 sec - exiting 11:20:18 (3412): No heartbeat from core client for 30 sec - exiting 11:20:19 (3412): No heartbeat from core client for 30 sec - exiting 11:20:20 (3412): No heartbeat from core client for 30 sec - exiting 11:20:21 (3412): No heartbeat from core client for 30 sec - exiting 11:20:22 (3412): No heartbeat from core client for 30 sec - exiting 11:20:23 (3412): No heartbeat from core client for 30 sec - exiting 11:20:24 (3412): No heartbeat from core client for 30 sec - exiting 11:20:25 (3412): No heartbeat from core client for 30 sec - exiting 11:20:26 (3412): No heartbeat from core client for 30 sec - exiting 11:20:27 (3412): No heartbeat from core client for 30 sec - exiting 11:20:28 (3412): No heartbeat from core client for 30 sec - exiting 11:20:29 (3412): No heartbeat from core client for 30 sec - exiting 11:20:30 (3412): No heartbeat from core client for 30 sec - exiting 11:20:31 (3412): No heartbeat from core client for 30 sec - exiting 11:20:32 (3412): No heartbeat from core client for 30 sec - exiting 11:20:33 (3412): No heartbeat from core client for 30 sec - exiting 11:20:34 (3412): No heartbeat from core client for 30 sec - exiting 11:20:35 (3412): No heartbeat from core client for 30 sec - exiting 11:20:36 (3412): No heartbeat from core client for 30 sec - exiting 11:20:37 (3412): No heartbeat from core client for 30 sec - exiting 11:20:38 (3412): No heartbeat from core client for 30 sec - exiting 11:20:40 (3412): No heartbeat from core client for 30 sec - exiting 11:20:41 (3412): No heartbeat from core client for 30 sec - exiting 11:20:42 (3412): No heartbeat from core client for 30 sec - exiting 11:20:43 (3412): No heartbeat from core client for 30 sec - exiting 11:20:44 (3412): No heartbeat from core client for 30 sec - exiting 11:20:45 (3412): No heartbeat from core client for 30 sec - exiting 11:20:46 (3412): No heartbeat from core client for 30 sec - exiting 11:20:47 (3412): No heartbeat from core client for 30 sec - exiting 11:20:48 (3412): No heartbeat from core client for 30 sec - exiting 11:20:49 (3412): No heartbeat from core client for 30 sec - exiting 11:20:50 (3412): No heartbeat from core client for 30 sec - exiting 11:20:51 (3412): No heartbeat from core client for 30 sec - exiting 11:20:52 (3412): No heartbeat from core client for 30 sec - exiting 11:20:53 (3412): No heartbeat from core client for 30 sec - exiting 11:20:54 (3412): No heartbeat from core client for 30 sec - exiting 11:20:55 (3412): No heartbeat from core client for 30 sec - exiting 11:20:56 (3412): No heartbeat from core client for 30 sec - exiting 11:20:57 (3412): No heartbeat from core client for 30 sec - exiting 11:20:58 (3412): No heartbeat from core client for 30 sec - exiting 11:20:59 (3412): No heartbeat from core client for 30 sec - exiting 11:21:00 (3412): No heartbeat from core client for 30 sec - exiting 11:21:01 (3412): No heartbeat from core client for 30 sec - exiting 11:21:02 (3412): No heartbeat from core client for 30 sec - exiting 11:21:03 (3412): No heartbeat from core client for 30 sec - exiting 11:21:04 (3412): No heartbeat from core client for 30 sec - exiting 11:21:05 (3412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=964, iMonCtr=2 09:47:29 (4080): No heartbeat from core client for 30 sec - exiting 09:47:30 (4080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:23:40 (4280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:24:51 (7816): No heartbeat from core client for 30 sec - exiting 20:24:52 (7816): No heartbeat from core client for 30 sec - exiting 20:24:53 (7816): No heartbeat from core client for 30 sec - exiting 20:24:54 (7816): No heartbeat from core client for 30 sec - exiting 20:24:55 (7816): No heartbeat from core client for 30 sec - exiting 20:24:56 (7816): No heartbeat from core client for 30 sec - exiting 20:24:57 (7816): No heartbeat from core client for 30 sec - exiting 20:24:58 (7816): No heartbeat from core client for 30 sec - exiting 20:24:59 (7816): No heartbeat from core client for 30 sec - exiting 20:25:00 (7816): No heartbeat from core client for 30 sec - exiting 20:25:01 (7816): No heartbeat from core client for 30 sec - exiting 20:25:02 (7816): No heartbeat from core client for 30 sec - exiting 20:25:03 (7816): No heartbeat from core client for 30 sec - exiting 20:25:04 (7816): No heartbeat from core client for 30 sec - exiting 20:25:05 (7816): No heartbeat from core client for 30 sec - exiting 20:25:06 (7816): No heartbeat from core client for 30 sec - exiting 20:25:07 (7816): No heartbeat from core client for 30 sec - exiting 20:25:08 (7816): No heartbeat from core client for 30 sec - exiting 20:25:09 (7816): No heartbeat from core client for 30 sec - exiting 20:25:10 (7816): No heartbeat from core client for 30 sec - exiting 20:25:11 (7816): No heartbeat from core client for 30 sec - exiting 20:25:12 (7816): No heartbeat from core client for 30 sec - exiting 20:25:13 (7816): No heartbeat from core client for 30 sec - exiting 20:25:14 (7816): No heartbeat from core client for 30 sec - exiting 20:25:15 (7816): No heartbeat from core client for 30 sec - exiting 20:25:16 (7816): No heartbeat from core client for 30 sec - exiting 20:25:17 (7816): No heartbeat from core client for 30 sec - exiting 20:25:18 (7816): No heartbeat from core client for 30 sec - exiting 20:25:19 (7816): No heartbeat from core client for 30 sec - exiting 20:25:20 (7816): No heartbeat from core client for 30 sec - exiting 20:25:21 (7816): No heartbeat from core client for 30 sec - exiting 20:25:22 (7816): No heartbeat from core client for 30 sec - exiting 07:14:03 (8208): No heartbeat from core client for 30 sec - exiting 07:14:04 (8208): No heartbeat from core client for 30 sec - exiting 07:14:05 (8208): No heartbeat from core client for 30 sec - exiting 07:14:06 (8208): No heartbeat from core client for 30 sec - exiting 07:14:07 (8208): No heartbeat from core client for 30 sec - exiting 07:14:08 (8208): No heartbeat from core client for 30 sec - exiting 07:14:09 (8208): No heartbeat from core client for 30 sec - exiting 07:14:10 (8208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8428, selfPID=8428, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 12:35:50 (7272): No heartbeat from core client for 30 sec - exiting 12:35:51 (7272): No heartbeat from core client for 30 sec - exiting 12:35:52 (7272): No heartbeat from core client for 30 sec - exiting 12:35:53 (7272): No heartbeat from core client for 30 sec - exiting 12:35:54 (7272): No heartbeat from core client for 30 sec - exiting 12:35:55 (7272): No heartbeat from core client for 30 sec - exiting 12:35:56 (7272): No heartbeat from core client for 30 sec - exiting 12:35:57 (7272): No heartbeat from core client for 30 sec - exiting 12:35:58 (7272): No heartbeat from core client for 30 sec - exiting 12:35:59 (7272): No heartbeat from core client for 30 sec - exiting 12:36:00 (7272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:09:45 (6844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=296, selfPID=296, iMonCtr=2 Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=1404, iMonCtr=1 Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6172, selfPID=7032, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 20:10:28 (7032): No heartbeat from core client for 30 sec - exiting 20:10:29 (7032): No heartbeat from core client for 30 sec - exiting 20:10:30 (7032): No heartbeat from core client for 30 sec - exiting 20:10:31 (7032): No heartbeat from core client for 30 sec - exiting 20:10:32 (7032): No heartbeat from core client for 30 sec - exiting 20:10:33 (7032): No heartbeat from core client for 30 sec - exiting 20:10:34 (7032): No heartbeat from core client for 30 sec - exiting 20:10:35 (7032): No heartbeat from core client for 30 sec - exiting 20:10:36 (7032): No heartbeat from core client for 30 sec - exiting Called boinc_finish 20:10:57 (7032): No heartbeat from core client for 30 sec - exiting 20:10:58 (7032): No heartbeat from core client for 30 sec - exiting </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_n8y3_2012_1_008599211_0_3.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_n8y3_2012_1_008599211_0_4.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_n8y3_2012_1_008599211_0_5.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_n8y3_2012_1_008599211_0_6.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_n8y3_2012_1_008599211_0_7.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_n8y3_2012_1_008599211_0_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_n8y3_2012_1_008599211_0_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_n8y3_2012_1_008599211_0_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_n8y3_2012_1_008599211_0_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_n8y3_2012_1_008599211_0_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Apr 2014 00:22:38	1315213	16416503	hadam3p_anz_n8y3_2012_1_008599211_0	23,339	273,211	11.7062
01 Apr 2014 01:36:22	1315213	16416503	hadam3p_anz_n8y3_2012_1_008599211_0	11,819	139,884	11.8355