Task 13251370

Name	hadam3p_saf_2c3j_1965_1_007403971_2
Workunit	7601401
Created	15 Aug 2011, 0:33:27 UTC
Sent	16 Aug 2011, 6:09:56 UTC
Report deadline	28 Jul 2012, 11:29:56 UTC
Received	3 Sep 2011, 7:23:15 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1156102
Run time	2 days 10 hours 39 min 4 sec
CPU time	15 hours 13 min 10 sec
Validate state	Invalid
Credit	1,309.70
Device peak FLOPS	2.55 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Southern Africa v6.09 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1956, selfPID=5592, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6088, iMonCtr=2 20:19:40 (4304): No heartbeat from core client for 30 sec - exiting 20:19:41 (4304): No heartbeat from core client for 30 sec - exiting 20:19:42 (4304): No heartbeat from core client for 30 sec - exiting 20:19:43 (4304): No heartbeat from core client for 30 sec - exiting 20:19:44 (4304): No heartbeat from core client for 30 sec - exiting 20:19:45 (4304): No heartbeat from core client for 30 sec - exiting 20:19:46 (4304): No heartbeat from core client for 30 sec - exiting 20:19:47 (4304): No heartbeat from core client for 30 sec - exiting 20:19:48 (4304): No heartbeat from core client for 30 sec - exiting 20:19:49 (4304): No heartbeat from core client for 30 sec - exiting 20:19:50 (4304): No heartbeat from core client for 30 sec - exiting 20:19:51 (4304): No heartbeat from core client for 30 sec - exiting 20:19:52 (4304): No heartbeat from core client for 30 sec - exiting 20:19:53 (4304): No heartbeat from core client for 30 sec - exiting 20:19:54 (4304): No heartbeat from core client for 30 sec - exiting 20:19:55 (4304): No heartbeat from core client for 30 sec - exiting 20:19:56 (4304): No heartbeat from core client for 30 sec - exiting 20:19:57 (4304): No heartbeat from core client for 30 sec - exiting 20:19:58 (4304): No heartbeat from core client for 30 sec - exiting 20:19:59 (4304): No heartbeat from core client for 30 sec - exiting 20:20:00 (4304): No heartbeat from core client for 30 sec - exiting 20:20:01 (4304): No heartbeat from core client for 30 sec - exiting 20:20:02 (4304): No heartbeat from core client for 30 sec - exiting 20:20:03 (4304): No heartbeat from core client for 30 sec - exiting 20:20:04 (4304): No heartbeat from core client for 30 sec - exiting 20:20:05 (4304): No heartbeat from core client for 30 sec - exiting 20:20:06 (4304): No heartbeat from core client for 30 sec - exiting 20:20:08 (4304): No heartbeat from core client for 30 sec - exiting 20:20:09 (4304): No heartbeat from core client for 30 sec - exiting 20:20:10 (4304): No heartbeat from core client for 30 sec - exiting 20:20:11 (4304): No heartbeat from core client for 30 sec - exiting 20:20:12 (4304): No heartbeat from core client for 30 sec - exiting 20:20:13 (4304): No heartbeat from core client for 30 sec - exiting 20:20:14 (4304): No heartbeat from core client for 30 sec - exiting 20:20:15 (4304): No heartbeat from core client for 30 sec - exiting 20:20:16 (4304): No heartbeat from core client for 30 sec - exiting 20:20:17 (4304): No heartbeat from core client for 30 sec - exiting 20:20:18 (4304): No heartbeat from core client for 30 sec - exiting 20:20:20 (4304): No heartbeat from core client for 30 sec - exiting 20:20:21 (4304): No heartbeat from core client for 30 sec - exiting 20:20:22 (4304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4288, selfPID=3340, iMonCtr=1 Model crash detected, will try to restart... 07:53:16 (4700): No heartbeat from core client for 30 sec - exiting 07:53:17 (4700): No heartbeat from core client for 30 sec - exiting 07:53:18 (4700): No heartbeat from core client for 30 sec - exiting 07:53:19 (4700): No heartbeat from core client for 30 sec - exiting 07:53:20 (4700): No heartbeat from core client for 30 sec - exiting 07:53:21 (4700): No heartbeat from core client for 30 sec - exiting 07:53:22 (4700): No heartbeat from core client for 30 sec - exiting 07:53:23 (4700): No heartbeat from core client for 30 sec - exiting 07:53:24 (4700): No heartbeat from core client for 30 sec - exiting 07:53:25 (4700): No heartbeat from core client for 30 sec - exiting 07:53:26 (4700): No heartbeat from core client for 30 sec - exiting 07:53:27 (4700): No heartbeat from core client for 30 sec - exiting 07:53:28 (4700): No heartbeat from core client for 30 sec - exiting 07:53:29 (4700): No heartbeat from core client for 30 sec - exiting 07:53:30 (4700): No heartbeat from core client for 30 sec - exiting 07:53:31 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:44:14 (4816): No heartbeat from core client for 30 sec - exiting 18:44:15 (4816): No heartbeat from core client for 30 sec - exiting 18:44:17 (4816): No heartbeat from core client for 30 sec - exiting 18:44:18 (4816): No heartbeat from core client for 30 sec - exiting 18:44:19 (4816): No heartbeat from core client for 30 sec - exiting 18:44:20 (4816): No heartbeat from core client for 30 sec - exiting 18:44:21 (4816): No heartbeat from core client for 30 sec - exiting 18:44:22 (4816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... C19:44:18 (4136): No heartbeat from core client for 30 sec - exiting 19:44:19 (4136): No heartbeat from core client for 30 sec - exiting 19:44:20 (4136): No heartbeat from core client for 30 sec - exiting 19:44:21 (4136): No heartbeat from core client for 30 sec - exiting 19:44:22 (4136): No heartbeat from core client for 30 sec - exiting 19:44:23 (4136): No heartbeat from core client for 30 sec - exiting 19:44:24 (4136): No heartbeat from core client for 30 sec - exiting 19:44:25 (4136): No heartbeat from core client for 30 sec - exiting 19:44:26 (4136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5236, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5244, selfPID=4328, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:21:43 (4224): No heartbeat from core client for 30 sec - exiting 08:21:44 (4224): No heartbeat from core client for 30 sec - exiting 08:21:45 (4224): No heartbeat from core client for 30 sec - exiting 08:21:46 (4224): No heartbeat from core client for 30 sec - exiting 08:21:47 (4224): No heartbeat from core client for 30 sec - exiting 08:21:48 (4224): No heartbeat from core client for 30 sec - exiting 08:21:49 (4224): No heartbeat from core client for 30 sec - exiting 08:21:50 (4224): No heartbeat from core client for 30 sec - exiting 08:21:51 (4224): No heartbeat from core client for 30 sec - exiting 08:21:52 (4224): No heartbeat from core client for 30 sec - exiting 08:21:53 (4224): No heartbeat from core client for 30 sec - exiting 08:21:54 (4224): No heartbeat from core client for 30 sec - exiting 08:21:55 (4224): No heartbeat from core client for 30 sec - exiting 08:21:56 (4224): No heartbeat from core client for 30 sec - exiting 08:21:57 (4224): No heartbeat from core client for 30 sec - exiting 08:21:58 (4224): No heartbeat from core client for 30 sec - exiting 08:21:59 (4224): No heartbeat from core client for 30 sec - exiting 08:22:00 (4224): No heartbeat from core client for 30 sec - exiting 08:22:01 (4224): No heartbeat from core client for 30 sec - exiting 08:22:02 (4224): No heartbeat from core client for 30 sec - exiting 08:22:03 (4224): No heartbeat from core client for 30 sec - exiting 08:22:04 (4224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_saf_2c3j_1965_1_007403971_2_8.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_2c3j_1965_1_007403971_2_9.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_2c3j_1965_1_007403971_2_10.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_2c3j_1965_1_007403971_2_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_2c3j_1965_1_007403971_2_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Sep 2011 21:01:20	1156102	13251370	hadam3p_saf_2c3j_1965_1_007403971_2	80,736	148,463	1.8389
01 Sep 2011 14:39:46	1156102	13251370	hadam3p_saf_2c3j_1965_1_007403971_2	69,216	126,373	1.8258
31 Aug 2011 19:55:34	1156102	13251370	hadam3p_saf_2c3j_1965_1_007403971_2	57,696	105,314	1.8253
31 Aug 2011 07:50:15	1156102	13251370	hadam3p_saf_2c3j_1965_1_007403971_2	46,176	84,137	1.8221
30 Aug 2011 17:56:22	1156102	13251370	hadam3p_saf_2c3j_1965_1_007403971_2	34,656	62,386	1.8002
30 Aug 2011 11:51:35	1156102	13251370	hadam3p_saf_2c3j_1965_1_007403971_2	23,136	41,423	1.7904
30 Aug 2011 06:16:42	1156102	13251370	hadam3p_saf_2c3j_1965_1_007403971_2	11,616	21,354	1.8383