Task 19219317

Name	hadam3p_anz_ka0g_201212_12_306_010273457_0
Workunit	10273457
Created	25 Jan 2016, 15:33:11 UTC
Sent	25 Jan 2016, 20:27:45 UTC
Report deadline	7 Jan 2017, 1:47:45 UTC
Received	31 Jan 2016, 22:05:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1382881
Run time	2 days 22 hours 27 min 36 sec
CPU time	2 days 19 hours 30 min 27 sec
Validate state	Invalid
Credit	2,000.18
Device peak FLOPS	3.15 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.6.22</core_client_version> <![CDATA[ <stderr_txt> 22:32:25 (9592): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:31:25 (4888): No heartbeat from core client for 30 sec - exiting 21:31:26 (4888): No heartbeat from core client for 30 sec - exiting 21:31:27 (4888): No heartbeat from core client for 30 sec - exiting 21:31:28 (4888): No heartbeat from core client for 30 sec - exiting 21:31:29 (4888): No heartbeat from core client for 30 sec - exiting 21:31:30 (4888): No heartbeat from core client for 30 sec - exiting 21:31:31 (4888): No heartbeat from core client for 30 sec - exiting 21:31:33 (4888): No heartbeat from core client for 30 sec - exiting 21:31:34 (4888): No heartbeat from core client for 30 sec - exiting 21:31:35 (4888): No heartbeat from core client for 30 sec - exiting 21:31:36 (4888): No heartbeat from core client for 30 sec - exiting 21:31:37 (4888): No heartbeat from core client for 30 sec - exiting 21:31:38 (4888): No heartbeat from core client for 30 sec - exiting 21:31:39 (4888): No heartbeat from core client for 30 sec - exiting 21:31:40 (4888): No heartbeat from core client for 30 sec - exiting 21:31:41 (4888): No heartbeat from core client for 30 sec - exiting 21:31:42 (4888): No heartbeat from core client for 30 sec - exiting 21:31:44 (4888): No heartbeat from core client for 30 sec - exiting 21:31:45 (4888): No heartbeat from core client for 30 sec - exiting 21:31:46 (4888): No heartbeat from core client for 30 sec - exiting 21:31:47 (4888): No heartbeat from core client for 30 sec - exiting 21:31:48 (4888): No heartbeat from core client for 30 sec - exiting 21:31:49 (4888): No heartbeat from core client for 30 sec - exiting 21:31:50 (4888): No heartbeat from core client for 30 sec - exiting 21:31:51 (4888): No heartbeat from core client for 30 sec - exiting 21:31:52 (4888): No heartbeat from core client for 30 sec - exiting 21:31:53 (4888): No heartbeat from core client for 30 sec - exiting 21:31:54 (4888): No heartbeat from core client for 30 sec - exiting 21:31:56 (4888): No heartbeat from core client for 30 sec - exiting 21:31:57 (4888): No heartbeat from core client for 30 sec - exiting 21:31:58 (4888): No heartbeat from core client for 30 sec - exiting 21:31:59 (4888): No heartbeat from core client for 30 sec - exiting 21:32:00 (4888): No heartbeat from core client for 30 sec - exiting 21:32:01 (4888): No heartbeat from core client for 30 sec - exiting 21:32:02 (4888): No heartbeat from core client for 30 sec - exiting 21:32:03 (4888): No heartbeat from core client for 30 sec - exiting 21:32:04 (4888): No heartbeat from core client for 30 sec - exiting 21:32:05 (4888): No heartbeat from core client for 30 sec - exiting 21:32:06 (4888): No heartbeat from core client for 30 sec - exiting 21:32:08 (4888): No heartbeat from core client for 30 sec - exiting 21:32:09 (4888): No heartbeat from core client for 30 sec - exiting 21:32:10 (4888): No heartbeat from core client for 30 sec - exiting 21:32:11 (4888): No heartbeat from core client for 30 sec - exiting 21:32:12 (4888): No heartbeat from core client for 30 sec - exiting 21:32:13 (4888): No heartbeat from core client for 30 sec - exiting 21:32:14 (4888): No heartbeat from core client for 30 sec - exiting 21:32:15 (4888): No heartbeat from core client for 30 sec - exiting 21:32:16 (4888): No heartbeat from core client for 30 sec - exiting 21:32:17 (4888): No heartbeat from core client for 30 sec - exiting 21:32:19 (4888): No heartbeat from core client for 30 sec - exiting 21:32:20 (4888): No heartbeat from core client for 30 sec - exiting 21:32:21 (4888): No heartbeat from core client for 30 sec - exiting 21:32:22 (4888): No heartbeat from core client for 30 sec - exiting 21:32:23 (4888): No heartbeat from core client for 30 sec - exiting 21:32:24 (4888): No heartbeat from core client for 30 sec - exiting 21:32:25 (4888): No heartbeat from core client for 30 sec - exiting 21:32:26 (4888): No heartbeat from core client for 30 sec - exiting 21:32:27 (4888): No heartbeat from core client for 30 sec - exiting 21:32:28 (4888): No heartbeat from core client for 30 sec - exiting 21:32:29 (4888): No heartbeat from core client for 30 sec - exiting 21:32:31 (4888): No heartbeat from core client for 30 sec - exiting 21:32:32 (4888): No heartbeat from core client for 30 sec - exiting 21:32:33 (4888): No heartbeat from core client for 30 sec - exiting 21:32:34 (4888): No heartbeat from core client for 30 sec - exiting 21:32:35 (4888): No heartbeat from core client for 30 sec - exiting 21:32:36 (4888): No heartbeat from core client for 30 sec - exiting 21:32:37 (4888): No heartbeat from core client for 30 sec - exiting 21:32:38 (9492): start_timer_thread(): CreateThread() failed, errno 0 21:32:38 (4888): No heartbeat from core client for 30 sec - exiting 21:32:39 (4888): No heartbeat from core client for 30 sec - exiting 21:32:39 (3460): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 21:32:43 (8168): start_timer_thread(): CreateThread() failed, errno 0 21:32:44 (3432): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 23:32:16 (4664): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:53:57 (5356): No heartbeat from core client for 30 sec - exiting 18:53:59 (5356): No heartbeat from core client for 30 sec - exiting 18:54:00 (5356): No heartbeat from core client for 30 sec - exiting 18:54:01 (5356): No heartbeat from core client for 30 sec - exiting 18:54:02 (5356): No heartbeat from core client for 30 sec - exiting 18:54:03 (5356): No heartbeat from core client for 30 sec - exiting 18:54:04 (5356): No heartbeat from core client for 30 sec - exiting 18:54:05 (5356): No heartbeat from core client for 30 sec - exiting 18:54:06 (5356): No heartbeat from core client for 30 sec - exiting 18:54:07 (5356): No heartbeat from core client for 30 sec - exiting 18:54:08 (5356): No heartbeat from core client for 30 sec - exiting 18:54:09 (5356): No heartbeat from core client for 30 sec - exiting 18:54:10 (5356): No heartbeat from core client for 30 sec - exiting 18:54:11 (5356): No heartbeat from core client for 30 sec - exiting 18:54:12 (5356): No heartbeat from core client for 30 sec - exiting 18:54:13 (5356): No heartbeat from core client for 30 sec - exiting 18:54:14 (5356): No heartbeat from core client for 30 sec - exiting 18:54:16 (5356): No heartbeat from core client for 30 sec - exiting 18:54:17 (5356): No heartbeat from core client for 30 sec - exiting 18:54:18 (5356): No heartbeat from core client for 30 sec - exiting 18:54:19 (5356): No heartbeat from core client for 30 sec - exiting 18:54:20 (5356): No heartbeat from core client for 30 sec - exiting 18:54:21 (5356): No heartbeat from core client for 30 sec - exiting 18:54:22 (5356): No heartbeat from core client for 30 sec - exiting 18:54:23 (5356): No heartbeat from core client for 30 sec - exiting 18:54:24 (5356): No heartbeat from core client for 30 sec - exiting 18:54:25 (5356): No heartbeat from core client for 30 sec - exiting 18:54:26 (5356): No heartbeat from core client for 30 sec - exiting 18:54:28 (5356): No heartbeat from core client for 30 sec - exiting 18:54:29 (5356): No heartbeat from core client for 30 sec - exiting 18:54:30 (5356): No heartbeat from core client for 30 sec - exiting 18:54:31 (5356): No heartbeat from core client for 30 sec - exiting 18:54:32 (5356): No heartbeat from core client for 30 sec - exiting 18:54:33 (5356): No heartbeat from core client for 30 sec - exiting 18:54:34 (5356): No heartbeat from core client for 30 sec - exiting 18:54:35 (5356): No heartbeat from core client for 30 sec - exiting 18:54:36 (5356): No heartbeat from core client for 30 sec - exiting 18:54:37 (5356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:06:40 (2972): No heartbeat from core client for 30 sec - exiting 03:06:42 (2972): No heartbeat from core client for 30 sec - exiting 03:06:43 (2972): No heartbeat from core client for 30 sec - exiting 03:06:44 (2972): No heartbeat from core client for 30 sec - exiting 03:06:45 (2972): No heartbeat from core client for 30 sec - exiting 03:06:46 (2972): No heartbeat from core client for 30 sec - exiting 03:06:47 (2972): No heartbeat from core client for 30 sec - exiting 03:06:48 (2972): No heartbeat from core client for 30 sec - exiting 03:06:49 (2972): No heartbeat from core client for 30 sec - exiting 03:06:50 (2972): No heartbeat from core client for 30 sec - exiting 03:06:52 (2972): No heartbeat from core client for 30 sec - exiting 03:06:53 (2972): No heartbeat from core client for 30 sec - exiting 03:06:54 (2972): No heartbeat from core client for 30 sec - exiting 03:06:55 (2972): No heartbeat from core client for 30 sec - exiting 03:06:56 (2972): No heartbeat from core client for 30 sec - exiting 03:06:57 (2972): No heartbeat from core client for 30 sec - exiting 03:06:58 (2972): No heartbeat from core client for 30 sec - exiting 03:06:59 (2972): No heartbeat from core client for 30 sec - exiting 03:07:00 (2972): No heartbeat from core client for 30 sec - exiting 03:07:01 (2972): No heartbeat from core client for 30 sec - exiting 03:07:02 (2972): No heartbeat from core client for 30 sec - exiting 03:07:04 (2972): No heartbeat from core client for 30 sec - exiting 03:07:05 (2972): No heartbeat from core client for 30 sec - exiting 03:07:06 (2972): No heartbeat from core client for 30 sec - exiting 03:07:07 (2972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:07:08 (2972): No heartbeat from core client for 30 sec - exiting 03:07:46 (800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:08:02 (5988): start_timer_thread(): CreateThread() failed, errno 0 03:08:04 (4480): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:56:24 (4960): No heartbeat from core client for 30 sec - exiting 18:56:25 (4960): No heartbeat from core client for 30 sec - exiting 18:56:26 (4960): No heartbeat from core client for 30 sec - exiting 18:56:27 (4960): No heartbeat from core client for 30 sec - exiting 18:56:28 (4960): No heartbeat from core client for 30 sec - exiting 18:56:30 (4960): No heartbeat from core client for 30 sec - exiting 18:56:31 (4960): No heartbeat from core client for 30 sec - exiting 18:56:32 (4960): No heartbeat from core client for 30 sec - exiting 18:56:33 (4960): No heartbeat from core client for 30 sec - exiting 18:56:34 (4960): No heartbeat from core client for 30 sec - exiting 18:56:35 (4960): No heartbeat from core client for 30 sec - exiting 18:56:36 (4960): No heartbeat from core client for 30 sec - exiting 18:56:37 (4960): No heartbeat from core client for 30 sec - exiting 18:56:38 (4960): No heartbeat from core client for 30 sec - exiting 18:56:39 (4960): No heartbeat from core client for 30 sec - exiting 18:56:41 (4960): No heartbeat from core client for 30 sec - exiting 18:56:42 (4960): No heartbeat from core client for 30 sec - exiting 18:56:43 (4960): No heartbeat from core client for 30 sec - exiting 18:56:44 (4960): No heartbeat from core client for 30 sec - exiting 18:56:45 (4960): No heartbeat from core client for 30 sec - exiting 18:56:46 (4960): No heartbeat from core client for 30 sec - exiting 18:56:47 (4960): No heartbeat from core client for 30 sec - exiting 18:56:48 (4960): No heartbeat from core client for 30 sec - exiting 18:56:49 (4960): No heartbeat from core client for 30 sec - exiting 18:56:50 (4960): No heartbeat from core client for 30 sec - exiting 18:56:51 (4960): No heartbeat from core client for 30 sec - exiting 18:56:53 (4960): No heartbeat from core client for 30 sec - exiting 18:56:54 (4960): No heartbeat from core client for 30 sec - exiting 18:56:55 (4960): No heartbeat from core client for 30 sec - exiting 18:56:56 (4960): No heartbeat from core client for 30 sec - exiting 18:56:57 (4960): No heartbeat from core client for 30 sec - exiting 18:56:58 (4960): No heartbeat from core client for 30 sec - exiting 18:56:59 (4960): No heartbeat from core client for 30 sec - exiting 18:57:00 (7692): start_timer_thread(): CreateThread() failed, errno 0 18:57:00 (4960): No heartbeat from core client for 30 sec - exiting 18:57:01 (8188): start_timer_thread(): CreateThread() failed, errno 0 18:57:01 (4960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:57:02 (4960): No heartbeat from core client for 30 sec - exiting 18:57:26 (2804): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... 20:43:07 (7348): No heartbeat from core client for 30 sec - exiting 20:43:08 (7348): No heartbeat from core client for 30 sec - exiting 20:43:09 (7348): No heartbeat from core client for 30 sec - exiting 20:43:10 (7348): No heartbeat from core client for 30 sec - exiting 20:43:11 (7348): No heartbeat from core client for 30 sec - exiting 20:43:12 (7348): No heartbeat from core client for 30 sec - exiting 20:43:13 (7348): No heartbeat from core client for 30 sec - exiting 20:43:14 (7348): No heartbeat from core client for 30 sec - exiting 20:43:16 (7348): No heartbeat from core client for 30 sec - exiting 20:43:17 (7348): No heartbeat from core client for 30 sec - exiting 20:43:18 (7348): No heartbeat from core client for 30 sec - exiting 20:43:19 (7348): No heartbeat from core client for 30 sec - exiting 20:43:20 (7348): No heartbeat from core client for 30 sec - exiting 20:43:21 (7348): No heartbeat from core client for 30 sec - exiting 20:43:22 (7348): No heartbeat from core client for 30 sec - exiting 20:43:23 (7348): No heartbeat from core client for 30 sec - exiting 20:43:24 (7348): No heartbeat from core client for 30 sec - exiting 20:43:25 (7348): No heartbeat from core client for 30 sec - exiting 20:43:27 (7348): No heartbeat from core client for 30 sec - exiting 20:43:28 (7348): No heartbeat from core client for 30 sec - exiting 20:43:29 (7348): No heartbeat from core client for 30 sec - exiting 20:43:30 (7348): No heartbeat from core client for 30 sec - exiting 20:43:31 (7348): No heartbeat from core client for 30 sec - exiting 20:43:32 (7348): No heartbeat from core client for 30 sec - exiting 20:43:33 (7348): No heartbeat from core client for 30 sec - exiting 20:43:34 (7348): No heartbeat from core client for 30 sec - exiting 20:43:35 (7348): No heartbeat from core client for 30 sec - exiting 20:43:36 (4432): start_timer_thread(): CreateThread() failed, errno 0 20:43:36 (7348): No heartbeat from core client for 30 sec - exiting 20:43:37 (9068): start_timer_thread(): CreateThread() failed, errno 0 20:43:37 (7348): No heartbeat from core client for 30 sec - exiting 20:43:39 (7348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:44:20 (5080): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:03:54 (8196): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:24:17 (6936): start_timer_thread(): CreateThread() failed, errno 0 22:24:18 (4924): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:52:37 (5736): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4784, selfPID=4784, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4784, selfPID=296, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_ka0g_201212_12_306_010273457_0_5.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_ka0g_201212_12_306_010273457_0_6.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_ka0g_201212_12_306_010273457_0_7.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_ka0g_201212_12_306_010273457_0_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_ka0g_201212_12_306_010273457_0_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_ka0g_201212_12_306_010273457_0_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_ka0g_201212_12_306_010273457_0_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_ka0g_201212_12_306_010273457_0_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Jan 2016 08:31:59	1382881	19219317	hadam3p_anz_ka0g_201212_12_306_010273457_0	46,379	236,665	5.1028
30 Jan 2016 07:35:28	1382881	19219317	hadam3p_anz_ka0g_201212_12_306_010273457_0	34,859	178,128	5.1100
28 Jan 2016 23:17:42	1382881	19219317	hadam3p_anz_ka0g_201212_12_306_010273457_0	23,339	118,854	5.0925
27 Jan 2016 03:05:34	1382881	19219317	hadam3p_anz_ka0g_201212_12_306_010273457_0	11,819	60,282	5.1004