Task 13689834

Name	hadam3p_saf_78k0_1999_1_007568025_1
Workunit	7746155
Created	2 Dec 2011, 15:57:31 UTC
Sent	16 Dec 2011, 0:31:51 UTC
Report deadline	27 Nov 2012, 5:51:51 UTC
Received	25 Dec 2011, 22:33:32 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1259863
Run time	3 days 18 hours 3 min 17 sec
CPU time	2 days 18 hours 57 min 59 sec
Validate state	Invalid
Credit	1,309.70
Device peak FLOPS	2.15 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Southern Africa v6.09 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3956, selfPID=5036, iMonCtr=1 Model crash detected, will try to restart... 19:48:22 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:48:24 (4588): No heartbeat from core client for 30 sec - exiting 19:48:25 (4588): No heartbeat from core client for 30 sec - exiting 19:48:26 (4588): No heartbeat from core client for 30 sec - exiting 19:48:27 (4588): No heartbeat from core client for 30 sec - exiting 19:48:28 (4588): No heartbeat from core client for 30 sec - exiting 19:48:29 (4588): No heartbeat from core client for 30 sec - exiting 19:48:30 (4588): No heartbeat from core client for 30 sec - exiting 19:48:31 (4588): No heartbeat from core client for 30 sec - exiting 19:48:32 (4588): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4972, selfPID=4588, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3160, selfPID=4524, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... CPDN Monitor - Quit request from BOINC... 19:59:48 (5116): No heartbeat from core client for 30 sec - exiting 19:59:49 (5116): No heartbeat from core client for 30 sec - exiting 19:59:50 (5116): No heartbeat from core client for 30 sec - exiting 19:59:51 (5116): No heartbeat from core client for 30 sec - exiting 19:59:52 (5116): No heartbeat from core client for 30 sec - exiting 19:59:53 (5116): No heartbeat from core client for 30 sec - exiting 19:59:54 (5116): No heartbeat from core client for 30 sec - exiting 19:59:55 (5116): No heartbeat from core client for 30 sec - exiting 19:59:56 (5116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3952, selfPID=4836, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:48:33 (4412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:48:34 (4412): No heartbeat from core client for 30 sec - exiting 19:48:35 (4412): No heartbeat from core client for 30 sec - exiting 19:48:36 (4412): No heartbeat from core client for 30 sec - exiting 19:48:37 (4412): No heartbeat from core client for 30 sec - exiting 19:48:38 (4412): No heartbeat from core client for 30 sec - exiting 19:48:39 (4412): No heartbeat from core client for 30 sec - exiting 19:48:40 (4412): No heartbeat from core client for 30 sec - exiting 19:48:41 (4412): No heartbeat from core client for 30 sec - exiting 19:48:42 (4412): No heartbeat from core client for 30 sec - exiting 19:48:43 (4412): No heartbeat from core client for 30 sec - exiting 19:48:44 (4412): No heartbeat from core client for 30 sec - exiting 19:48:45 (4412): No heartbeat from core client for 30 sec - exiting 19:48:46 (4412): No heartbeat from core client for 30 sec - exiting 19:48:47 (4412): No heartbeat from core client for 30 sec - exiting 19:48:48 (4412): No heartbeat from core client for 30 sec - exiting 19:48:49 (4412): No heartbeat from core client for 30 sec - exiting 19:48:50 (4412): No heartbeat from core client for 30 sec - exiting 19:48:51 (4412): No heartbeat from core client for 30 sec - exiting 19:48:52 (4412): No heartbeat from core client for 30 sec - exiting 19:48:53 (4412): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4176, iMonCtr=2 Model crash detected, will try to restart... 23:53:09 (5000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:53:11 (5000): No heartbeat from core client for 30 sec - exiting 19:06:22 (4940): No heartbeat from core client for 30 sec - exiting 19:06:24 (4940): No heartbeat from core client for 30 sec - exiting 19:06:25 (4940): No heartbeat from core client for 30 sec - exiting 19:06:26 (4940): No heartbeat from core client for 30 sec - exiting 19:07:00 (4940): No heartbeat from core client for 30 sec - exiting 19:07:01 (4940): No heartbeat from core client for 30 sec - exiting 19:07:02 (4940): No heartbeat from core client for 30 sec - exiting 19:07:03 (4940): No heartbeat from core client for 30 sec - exiting 19:07:04 (4940): No heartbeat from core client for 30 sec - exiting 19:07:05 (4940): No heartbeat from core client for 30 sec - exiting 19:07:06 (4940): No heartbeat from core client for 30 sec - exiting 19:07:07 (4940): No heartbeat from core client for 30 sec - exiting 19:07:08 (4940): No heartbeat from core client for 30 sec - exiting 19:07:09 (4940): No heartbeat from core client for 30 sec - exiting 19:07:10 (4940): No heartbeat from core client for 30 sec - exiting 19:07:12 (4940): No heartbeat from core client for 30 sec - exiting 19:07:13 (4940): No heartbeat from core client for 30 sec - exiting 19:07:14 (4940): No heartbeat from core client for 30 sec - exiting 19:07:15 (4940): No heartbeat from core client for 30 sec - exiting 19:07:16 (4940): No heartbeat from core client for 30 sec - exiting 19:07:17 (4940): No heartbeat from core client for 30 sec - exiting 19:07:18 (4940): No heartbeat from core client for 30 sec - exiting 19:07:19 (4940): No heartbeat from core client for 30 sec - exiting 19:07:20 (4940): No heartbeat from core client for 30 sec - exiting 19:07:21 (4940): No heartbeat from core client for 30 sec - exiting 19:07:22 (4940): No heartbeat from core client for 30 sec - exiting 19:07:24 (4940): No heartbeat from core client for 30 sec - exiting 19:07:25 (4940): No heartbeat from core client for 30 sec - exiting 19:07:26 (4940): No heartbeat from core client for 30 sec - exiting 19:07:27 (4940): No heartbeat from core client for 30 sec - exiting 19:07:28 (4940): No heartbeat from core client for 30 sec - exiting 19:07:29 (4940): No heartbeat from core client for 30 sec - exiting 19:07:30 (4940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3536, selfPID=4276, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5056, selfPID=5056, iMonCtr=2 16:43:07 (4008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:18:42 (3912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:18:44 (3912): No heartbeat from core client for 30 sec - exiting 17:18:45 (3912): No heartbeat from core client for 30 sec - exiting 17:18:46 (3912): No heartbeat from core client for 30 sec - exiting 17:18:47 (3912): No heartbeat from core client for 30 sec - exiting 17:18:48 (3912): No heartbeat from core client for 30 sec - exiting 17:18:49 (3912): No heartbeat from core client for 30 sec - exiting 17:18:50 (3912): No heartbeat from core client for 30 sec - exiting 17:18:51 (3912): No heartbeat from core client for 30 sec - exiting 17:18:52 (3912): No heartbeat from core client for 30 sec - exiting 17:19:02 (3912): No heartbeat from core client for 30 sec - exiting 17:19:03 (3912): No heartbeat from core client for 30 sec - exiting 17:19:05 (3912): No heartbeat from core client for 30 sec - exiting 17:19:06 (3912): No heartbeat from core client for 30 sec - exiting 17:19:10 (3912): No heartbeat from core client for 30 sec - exiting 17:19:11 (3912): No heartbeat from core client for 30 sec - exiting 17:19:13 (3912): No heartbeat from core client for 30 sec - exiting 17:19:14 (3912): No heartbeat from core client for 30 sec - exiting 17:19:15 (3912): No heartbeat from core client for 30 sec - exiting 17:19:16 (3912): No heartbeat from core client for 30 sec - exiting 17:19:17 (3912): No heartbeat from core client for 30 sec - exiting 17:19:18 (3912): No heartbeat from core client for 30 sec - exiting 17:19:19 (3912): No heartbeat from core client for 30 sec - exiting 17:19:20 (3912): No heartbeat from core client for 30 sec - exiting 17:19:21 (3912): No heartbeat from core client for 30 sec - exiting 17:19:22 (3912): No heartbeat from core client for 30 sec - exiting 17:19:23 (3912): No heartbeat from core client for 30 sec - exiting 17:19:33 (3912): No heartbeat from core client for 30 sec - exiting 17:19:34 (3912): No heartbeat from core client for 30 sec - exiting 17:19:35 (3912): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4680, selfPID=3856, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 19:15:03 (1224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:15:04 (1224): No heartbeat from core client for 30 sec - exiting 19:15:06 (1224): No heartbeat from core client for 30 sec - exiting 19:15:07 (1224): No heartbeat from core client for 30 sec - exiting 19:15:08 (1224): No heartbeat from core client for 30 sec - exiting 19:15:09 (1224): No heartbeat from core client for 30 sec - exiting 19:15:10 (1224): No heartbeat from core client for 30 sec - exiting 19:15:11 (1224): No heartbeat from core client for 30 sec - exiting 19:15:12 (1224): No heartbeat from core client for 30 sec - exiting 19:15:13 (1224): No heartbeat from core client for 30 sec - exiting 19:15:14 (1224): No heartbeat from core client for 30 sec - exiting 19:15:15 (1224): No heartbeat from core client for 30 sec - exiting 19:15:16 (1224): No heartbeat from core client for 30 sec - exiting 19:15:18 (1224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 20:35:23 (668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:35:24 (668): No heartbeat from core client for 30 sec - exiting 20:35:25 (668): No heartbeat from core client for 30 sec - exiting 20:35:26 (668): No heartbeat from core client for 30 sec - exiting 20:35:27 (668): No heartbeat from core client for 30 sec - exiting 20:35:28 (668): No heartbeat from core client for 30 sec - exiting 20:35:29 (668): No heartbeat from core client for 30 sec - exiting 20:35:30 (668): No heartbeat from core client for 30 sec - exiting 20:35:31 (668): No heartbeat from core client for 30 sec - exiting 20:35:32 (668): No heartbeat from core client for 30 sec - exiting 20:35:34 (668): No heartbeat from core client for 30 sec - exiting 20:35:35 (668): No heartbeat from core client for 30 sec - exiting 20:35:36 (668): No heartbeat from core client for 30 sec - exiting 20:35:37 (668): No heartbeat from core client for 30 sec - exiting 20:35:38 (668): No heartbeat from core client for 30 sec - exiting 20:35:39 (668): No heartbeat from core client for 30 sec - exiting 20:35:40 (668): No heartbeat from core client for 30 sec - exiting 20:35:41 (668): No heartbeat from core client for 30 sec - exiting 20:35:42 (668): No heartbeat from core client for 30 sec - exiting 20:35:43 (668): No heartbeat from core client for 30 sec - exiting 20:35:44 (668): No heartbeat from core client for 30 sec - exiting 20:45:25 (3268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:15:48 (3932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:15:50 (3932): No heartbeat from core client for 30 sec - exiting 18:15:51 (3932): No heartbeat from core client for 30 sec - exiting 18:15:52 (3932): No heartbeat from core client for 30 sec - exiting 18:15:53 (3932): No heartbeat from core client for 30 sec - exiting 18:15:54 (3932): No heartbeat from core client for 30 sec - exiting 18:15:55 (3932): No heartbeat from core client for 30 sec - exiting 18:15:56 (3932): No heartbeat from core client for 30 sec - exiting 18:15:57 (3932): No heartbeat from core client for 30 sec - exiting 18:15:58 (3932): No heartbeat from core client for 30 sec - exiting 18:15:59 (3932): No heartbeat from core client for 30 sec - exiting 18:16:00 (3932): No heartbeat from core client for 30 sec - exiting 18:16:02 (3932): No heartbeat from core client for 30 sec - exiting 18:16:03 (3932): No heartbeat from core client for 30 sec - exiting 18:26:45 (668): Can't acquire lockfile (32) - waiting 35s 18:27:20 (668): Can't acquire lockfile (32) - exiting 18:27:20 (668): Error: vZXÍt@CÉANZXÅ«Ü¹ñBÊÌvZXªgpÅ·B (0x20) 19:01:32 (3760): No heartbeat from core client for 30 sec - exiting 19:01:33 (3760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4456, selfPID=1564, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4492, selfPID=3908, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4808, selfPID=2276, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_saf_78k0_1999_1_007568025_1_8.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_78k0_1999_1_007568025_1_9.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_78k0_1999_1_007568025_1_10.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_78k0_1999_1_007568025_1_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_78k0_1999_1_007568025_1_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Dec 2011 03:32:38	1186039	13689834	hadam3p_saf_78k0_1999_1_007568025_1	80,736	217,523	2.6943
24 Dec 2011 05:14:18	1186039	13689834	hadam3p_saf_78k0_1999_1_007568025_1	69,216	187,134	2.7036
21 Dec 2011 13:39:14	1186039	13689834	hadam3p_saf_78k0_1999_1_007568025_1	57,696	156,265	2.7084
19 Dec 2011 13:43:50	1186039	13689834	hadam3p_saf_78k0_1999_1_007568025_1	46,176	124,230	2.6904
18 Dec 2011 00:15:49	925923	13689834	hadam3p_saf_78k0_1999_1_007568025_1	34,656	92,586	2.6716
17 Dec 2011 04:07:48	925923	13689834	hadam3p_saf_78k0_1999_1_007568025_1	23,136	62,001	2.6798
16 Dec 2011 18:51:50	925923	13689834	hadam3p_saf_78k0_1999_1_007568025_1	11,616	31,492	2.7111