Task 14169903

Name	hadam3p_eu_9630_1968_1_007757742_1
Workunit	7912851
Created	21 Feb 2012, 20:21:46 UTC
Sent	21 Feb 2012, 20:41:55 UTC
Report deadline	3 Feb 2013, 2:01:55 UTC
Received	11 Apr 2012, 19:10:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1129407
Run time	7 days 10 hours 42 min 52 sec
CPU time	4 days 17 hours 57 min 49 sec
Validate state	Invalid
Credit	995.30
Device peak FLOPS	2.58 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Europe v6.09 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2100, selfPID=2040, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2800, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=152, selfPID=1496, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1824, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2512, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2520, selfPID=1256, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=988, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2608, selfPID=1408, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2844, selfPID=952, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2128, selfPID=1820, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3384, iMonCtr=2 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=768, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2056, selfPID=1268, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2052, selfPID=1820, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3804, selfPID=1112, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2348, selfPID=1436, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1800, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2780, selfPID=1816, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process Gs nolobal Worker:: CPDN process is not running, exiting, bRetVal = 1, chec Model crash detected, will try t o restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3904, selfPID=1468, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1756, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1824, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=368, selfPID=1472, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3668, selfPID=1428, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1472, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2724, selfPID=716, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5560, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5568, selfPID=1224, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3104, selfPID=3296, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3504, selfPID=1384, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2828, selfPID=3272, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4064, selfPID=3736, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2944, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2788, selfPID=4056, iMonCtr=1 Model crash detected, will try to restart... CGntroller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3128, iMonCtr=2 Model crash detected, will try to restart... lobal Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5332, selfPID=1976, iMonCtr=1 Model crash detected, will try to restart... GSuspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4064, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3784, selfPID=2604, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2532, selfPID=2704, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3548, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1172, selfPID=2900, iMonCtr=1 Model crash detected, will try to restart... GController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5264, selfPID=3144, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1636, selfPID=2484, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=804, selfPID=3788, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2208, selfPID=2832, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2040, selfPID=3664, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2516, selfPID=2956, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3696, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=108, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4596, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3500, selfPID=3508, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4064, selfPID=3484, iMonCtr=1 Model crash detected, will try to restart... 09:11:47 (3624): No heartbeat from core client for 30 sec - exiting 09:11:48 (3624): No heartbeat from core client for 30 sec - exiting 09:11:49 (3624): No heartbeat from core client for 30 sec - exiting 09:11:50 (3624): No heartbeat from core client for 30 sec - exiting 09:11:51 (3624): No heartbeat from core client for 30 sec - exiting 09:11:52 (3624): No heartbeat from core client for 30 sec - exiting 09:11:53 (3624): No heartbeat from core client for 30 sec - exiting 09:11:54 (3624): No heartbeat from core client for 30 sec - exiting 09:11:55 (3624): No heartbeat from core client for 30 sec - exiting 09:11:56 (3624): No heartbeat from core client for 30 sec - exiting 09:11:57 (3624): No heartbeat from core client for 30 sec - exiting 09:11:58 (3624): No heartbeat from core client for 30 sec - exiting 09:11:59 (3624): No heartbeat from core client for 30 sec - exiting 09:12:00 (3624): No heartbeat from core client for 30 sec - exiting 09:12:01 (3624): No heartbeat from core client for 30 sec - exiting 09:12:02 (3624): No heartbeat from core client for 30 sec - exiting 09:12:03 (3624): No heartbeat from core client for 30 sec - exiting 09:12:04 (3624): No heartbeat from core client for 30 sec - exiting 09:12:05 (3624): No heartbeat from core client for 30 sec - exiting 09:12:06 (3624): No heartbeat from core client for 30 sec - exiting 09:12:07 (3624): No heartbeat from core client for 30 sec - exiting 09:12:08 (3624): No heartbeat from core client for 30 sec - exiting 09:12:09 (3624): No heartbeat from core client for 30 sec - exiting 09:12:10 (3624): No heartbeat from core client for 30 sec - exiting 09:12:11 (3624): No heartbeat from core client for 30 sec - exiting 09:12:12 (3624): No heartbeat from core client for 30 sec - exiting 09:12:13 (3624): No heartbeat from core client for 30 sec - exiting 09:12:14 (3624): No heartbeat from core client for 30 sec - exiting 09:12:15 (3624): No heartbeat from core client for 30 sec - exiting 09:12:16 (3624): No heartbeat from core client for 30 sec - exiting 09:12:17 (3624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:57:56 (2672): No heartbeat from core client for 30 sec - exiting 11:57:57 (2672): No heartbeat from core client for 30 sec - exiting 11:57:58 (2672): No heartbeat from core client for 30 sec - exiting 11:57:59 (2672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3608, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1948, selfPID=2996, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2720, selfPID=244, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... CGlobal Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2820, iMonCtr=2 ontroller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3476, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2564, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2132, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> <file_xfer_error> <file_name>hadam3p_eu_9630_1968_1_007757742_1_6.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_9630_1968_1_007757742_1_7.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_9630_1968_1_007757742_1_8.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_9630_1968_1_007757742_1_9.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_9630_1968_1_007757742_1_10.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_9630_1968_1_007757742_1_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_9630_1968_1_007757742_1_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Apr 2012 19:35:10	1129407	14169903	hadam3p_eu_9630_1968_1_007757742_1	57,696	398,169	6.9012
03 Apr 2012 22:21:01	1129407	14169903	hadam3p_eu_9630_1968_1_007757742_1	46,176	318,184	6.8907
28 Mar 2012 19:33:48	1129407	14169903	hadam3p_eu_9630_1968_1_007757742_1	34,657	238,954	6.8948
27 Mar 2012 21:28:24	1129407	14169903	hadam3p_eu_9630_1968_1_007757742_1	34,656	237,946	6.8659
17 Mar 2012 13:34:29	1129407	14169903	hadam3p_eu_9630_1968_1_007757742_1	23,138	159,138	6.8778
17 Mar 2012 10:10:39	1129407	14169903	hadam3p_eu_9630_1968_1_007757742_1	23,136	158,074	6.8324
02 Mar 2012 20:58:16	1129407	14169903	hadam3p_eu_9630_1968_1_007757742_1	11,616	79,733	6.8641