Task 13195349

Name	hadam3p_eu_2u33_1976_1_007385233_1
Workunit	7582663
Created	3 Aug 2011, 1:09:48 UTC
Sent	3 Aug 2011, 1:19:27 UTC
Report deadline	15 Jul 2012, 6:39:27 UTC
Received	26 Aug 2011, 1:21:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	890420
Run time	6 days 20 hours 9 min 10 sec
CPU time	4 days 7 hours 24 min 42 sec
Validate state	Invalid
Credit	1,988.94
Device peak FLOPS	2.26 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Europe v6.09 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2872, selfPID=1904, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4468, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4876, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4116, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2440, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5356, selfPID=4228, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5448, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5460, selfPID=1656, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 09:30:31 (4960): No heartbeat from core client for 30 sec - exiting 09:30:32 (4960): No heartbeat from core client for 30 sec - exiting 09:30:33 (4960): No heartbeat from core client for 30 sec - exiting 09:30:34 (4960): No heartbeat from core client for 30 sec - exiting 09:30:35 (4960): No heartbeat from core client for 30 sec - exiting 09:30:36 (4960): No heartbeat from core client for 30 sec - exiting 09:30:37 (4960): No heartbeat from core client for 30 sec - exiting 09:30:38 (4960): No heartbeat from core client for 30 sec - exiting 09:30:39 (4960): No heartbeat from core client for 30 sec - exiting 09:30:40 (4960): No heartbeat from core client for 30 sec - exiting 09:30:41 (4960): No heartbeat from core client for 30 sec - exiting 09:30:42 (4960): No heartbeat from core client for 30 sec - exiting 09:30:43 (4960): No heartbeat from core client for 30 sec - exiting 09:30:44 (4960): No heartbeat from core client for 30 sec - exiting 09:30:45 (4960): No heartbeat from core client for 30 sec - exiting 09:30:46 (4960): No heartbeat from core client for 30 sec - exiting 09:30:47 (4960): No heartbeat from core client for 30 sec - exiting 09:30:48 (4960): No heartbeat from core client for 30 sec - exiting 09:30:49 (4960): No heartbeat from core client for 30 sec - exiting 09:30:50 (4960): No heartbeat from core client for 30 sec - exiting 09:30:51 (4960): No heartbeat from core client for 30 sec - exiting 09:30:52 (4960): No heartbeat from core client for 30 sec - exiting 09:30:53 (4960): No heartbeat from core client for 30 sec - exiting 09:30:54 (4960): No heartbeat from core client for 30 sec - exiting 09:30:55 (4960): No heartbeat from core client for 30 sec - exiting 09:30:56 (4960): No heartbeat from core client for 30 sec - exiting 09:30:57 (4960): No heartbeat from core client for 30 sec - exiting 09:30:58 (4960): No heartbeat from core client for 30 sec - exiting 09:30:59 (4960): No heartbeat from core client for 30 sec - exiting 09:31:00 (4960): No heartbeat from core client for 30 sec - exiting 09:31:01 (4960): No heartbeat from core client for 30 sec - exiting 09:31:02 (4960): No heartbeat from core client for 30 sec - exiting 09:31:03 (4960): No heartbeat from core client for 30 sec - exiting 09:31:04 (4960): No heartbeat from core client for 30 sec - exiting 09:31:05 (4960): No heartbeat from core client for 30 sec - exiting 09:31:06 (4960): No heartbeat from core client for 30 sec - exiting 09:31:07 (4960): No heartbeat from core client for 30 sec - exiting 09:31:08 (4960): No heartbeat from core client for 30 sec - exiting 09:31:09 (4960): No heartbeat from core client for 30 sec - exiting 09:31:10 (4960): No heartbeat from core client for 30 sec - exiting 09:31:11 (4960): No heartbeat from core client for 30 sec - exiting 09:31:12 (4960): No heartbeat from core client for 30 sec - exiting 09:31:13 (4960): No heartbeat from core client for 30 sec - exiting 09:31:14 (4960): No heartbeat from core client for 30 sec - exiting 09:31:15 (4960): No heartbeat from core client for 30 sec - exiting 09:31:16 (4960): No heartbeat from core client for 30 sec - exiting 09:31:17 (4960): No heartbeat from core client for 30 sec - exiting 09:31:18 (4960): No heartbeat from core client for 30 sec - exiting 09:31:19 (4960): No heartbeat from core client for 30 sec - exiting 09:31:20 (4960): No heartbeat from core client for 30 sec - exiting 09:31:21 (4960): No heartbeat from core client for 30 sec - exiting 09:31:22 (4960): No heartbeat from core client for 30 sec - exiting 09:31:23 (4960): No heartbeat from core client for 30 sec - exiting 09:31:24 (4960): No heartbeat from core client for 30 sec - exiting 09:31:25 (4960): No heartbeat from core client for 30 sec - exiting 09:31:26 (4960): No heartbeat from core client for 30 sec - exiting 09:31:27 (4960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5712, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5768, iMonCtr=2 Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4108, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=160, iMonCtr=2 Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4368, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4948, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4324, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5480, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4592, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5272, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4664, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4216, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN processrocess is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4396, iMonCtr=2 is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4244, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4912, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4704, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5432, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 09:18:55 (5148): No heartbeat from core client for 30 sec - exiting 09:18:56 (5148): No heartbeat from core client for 30 sec - exiting 09:18:57 (5148): No heartbeat from core client for 30 sec - exiting 09:18:58 (5148): No heartbeat from core client for 30 sec - exiting 09:18:59 (5148): No heartbeat from core client for 30 sec - exiting 09:19:00 (5148): No heartbeat from core client for 30 sec - exiting 09:19:01 (5148): No heartbeat from core client for 30 sec - exiting 09:19:02 (5148): No heartbeat from core client for 30 sec - exiting 09:19:03 (5148): No heartbeat from core client for 30 sec - exiting 09:19:04 (5148): No heartbeat from core client for 30 sec - exiting 09:19:05 (5148): No heartbeat from core client for 30 sec - exiting 09:19:06 (5148): No heartbeat from core client for 30 sec - exiting 09:19:07 (5148): No heartbeat from core client for 30 sec - exiting 09:19:08 (5148): No heartbeat from core client for 30 sec - exiting 09:19:09 (5148): No heartbeat from core client for 30 sec - exiting 09:19:10 (5148): No heartbeat from core client for 30 sec - exiting 09:19:11 (5148): No heartbeat from core client for 30 sec - exiting 09:19:12 (5148): No heartbeat from core client for 30 sec - exiting 09:19:13 (5148): No heartbeat from core client for 30 sec - exiting 09:19:14 (5148): No heartbeat from core client for 30 sec - exiting 09:19:15 (5148): No heartbeat from core client for 30 sec - exiting 09:19:16 (5148): No heartbeat from core client for 30 sec - exiting 09:19:17 (5148): No heartbeat from core client for 30 sec - exiting 09:19:18 (5148): No heartbeat from core client for 30 sec - exiting 09:19:19 (5148): No heartbeat from core client for 30 sec - exiting 09:19:20 (5148): No heartbeat from core client for 30 sec - exiting 09:19:21 (5148): No heartbeat from core client for 30 sec - exiting 09:19:22 (5148): No heartbeat from core client for 30 sec - exiting 09:19:23 (5148): No heartbeat from core client for 30 sec - exiting 09:19:24 (5148): No heartbeat from core client for 30 sec - exiting 09:19:25 (5148): No heartbeat from core client for 30 sec - exiting 09:19:26 (5148): No heartbeat from core client for 30 sec - exiting 09:19:27 (5148): No heartbeat from core client for 30 sec - exiting 09:19:28 (5148): No heartbeat from core client for 30 sec - exiting 09:19:29 (5148): No heartbeat from core client for 30 sec - exiting 09:19:30 (5148): No heartbeat from core client for 30 sec - exiting 09:19:31 (5148): No heartbeat from core client for 30 sec - exiting 09:19:32 (5148): No heartbeat from core client for 30 sec - exiting 09:19:33 (5148): No heartbeat from core client for 30 sec - exiting 09:19:34 (5148): No heartbeat from core client for 30 sec - exiting 09:19:35 (5148): No heartbeat from core client for 30 sec - exiting 09:19:36 (5148): No heartbeat from core client for 30 sec - exiting 09:19:37 (5148): No heartbeat from core client for 30 sec - exiting 09:19:38 (5148): No heartbeat from core client for 30 sec - exiting 09:19:39 (5148): No heartbeat from core client for 30 sec - exiting 09:19:40 (5148): No heartbeat from core client for 30 sec - exiting 09:19:41 (5148): No heartbeat from core client for 30 sec - exiting 09:19:42 (5148): No heartbeat from core client for 30 sec - exiting 09:19:43 (5148): No heartbeat from core client for 30 sec - exiting 09:19:44 (5148): No heartbeat from core client for 30 sec - exiting 09:19:45 (5148): No heartbeat from core client for 30 sec - exiting 09:19:46 (5148): No heartbeat from core client for 30 sec - exiting 09:19:47 (5148): No heartbeat from core client for 30 sec - exiting 09:19:48 (5148): No heartbeat from core client for 30 sec - exiting 09:19:49 (5148): No heartbeat from core client for 30 sec - exiting 09:19:50 (5148): No heartbeat from core client for 30 sec - exiting 09:19:51 (5148): No heartbeat from core client for 30 sec - exiting 09:19:52 (5148): No heartbeat from core client for 30 sec - exiting 09:19:53 (5148): No heartbeat from core client for 30 sec - exiting 09:19:54 (5148): No heartbeat from core client for 30 sec - exiting 09:19:55 (5148): No heartbeat from core client for 30 sec - exiting 09:19:56 (5148): No heartbeat from core client for 30 sec - exiting 09:19:57 (5148): No heartbeat from core client for 30 sec - exiting 09:19:58 (5148): No heartbeat from core client for 30 sec - exiting 09:19:59 (5148): No heartbeat from core client for 30 sec - exiting 09:20:00 (5148): No heartbeat from core client for 30 sec - exiting 09:20:01 (5148): No heartbeat from core client for 30 sec - exiting 09:20:02 (5148): No heartbeat from core client for 30 sec - exiting 09:20:03 (5148): No heartbeat from core client for 30 sec - exiting 09:20:04 (5148): No heartbeat from core client for 30 sec - exiting 09:20:05 (5148): No heartbeat from core client for 30 sec - exiting 09:20:06 (5148): No heartbeat from core client for 30 sec - exiting 09:20:07 (5148): No heartbeat from core client for 30 sec - exiting 09:20:08 (5148): No heartbeat from core client for 30 sec - exiting 09:20:09 (5148): No heartbeat from core client for 30 sec - exiting 09:20:10 (5148): No heartbeat from core client for 30 sec - exiting 09:20:11 (5148): No heartbeat from core client for 30 sec - exiting 09:20:12 (5148): No heartbeat from core client for 30 sec - exiting 09:20:13 (5148): No heartbeat from core client for 30 sec - exiting 09:20:14 (5148): No heartbeat from core client for 30 sec - exiting 09:20:15 (5148): No heartbeat from core client for 30 sec - exiting 09:20:16 (5148): No heartbeat from core client for 30 sec - exiting 09:20:17 (5148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5828, selfPID=4256, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 09:14:20 (4976): No heartbeat from core client for 30 sec - exiting 09:14:21 (4976): No heartbeat from core client for 30 sec - exiting 09:14:22 (4976): No heartbeat from core client for 30 sec - exiting 09:14:23 (4976): No heartbeat from core client for 30 sec - exiting 09:14:24 (4976): No heartbeat from core client for 30 sec - exiting 09:14:25 (4976): No heartbeat from core client for 30 sec - exiting 09:14:26 (4976): No heartbeat from core client for 30 sec - exiting 09:14:27 (4976): No heartbeat from core client for 30 sec - exiting 09:14:28 (4976): No heartbeat from core client for 30 sec - exiting 09:14:29 (4976): No heartbeat from core client for 30 sec - exiting 09:14:30 (4976): No heartbeat from core client for 30 sec - exiting 09:14:31 (4976): No heartbeat from core client for 30 sec - exiting 09:14:32 (4976): No heartbeat from core client for 30 sec - exiting 09:14:33 (4976): No heartbeat from core client for 30 sec - exiting 09:14:34 (4976): No heartbeat from core client for 30 sec - exiting 09:14:35 (4976): No heartbeat from core client for 30 sec - exiting 09:14:36 (4976): No heartbeat from core client for 30 sec - exiting 09:14:37 (4976): No heartbeat from core client for 30 sec - exiting 09:14:38 (4976): No heartbeat from core client for 30 sec - exiting 09:14:39 (4976): No heartbeat from core client for 30 sec - exiting 09:14:40 (4976): No heartbeat from core client for 30 sec - exiting 09:14:41 (4976): No heartbeat from core client for 30 sec - exiting 09:14:42 (4976): No heartbeat from core client for 30 sec - exiting 09:14:43 (4976): No heartbeat from core client for 30 sec - exiting 09:14:44 (4976): No heartbeat from core client for 30 sec - exiting 09:14:45 (4976): No heartbeat from core client for 30 sec - exiting 09:14:46 (4976): No heartbeat from core client for 30 sec - exiting 09:14:47 (4976): No heartbeat from core client for 30 sec - exiting 09:14:48 (4976): No heartbeat from core client for 30 sec - exiting 09:14:49 (4976): No heartbeat from core client for 30 sec - exiting 09:14:50 (4976): No heartbeat from core client for 30 sec - exiting 09:14:51 (4976): No heartbeat from core client for 30 sec - exiting 09:14:52 (4976): No heartbeat from core client for 30 sec - exiting 09:14:53 (4976): No heartbeat from core client for 30 sec - exiting 09:14:54 (4976): No heartbeat from core client for 30 sec - exiting 09:14:55 (4976): No heartbeat from core client for 30 sec - exiting 09:14:56 (4976): No heartbeat from core client for 30 sec - exiting 09:14:57 (4976): No heartbeat from core client for 30 sec - exiting 09:14:58 (4976): No heartbeat from core client for 30 sec - exiting 09:14:59 (4976): No heartbeat from core client for 30 sec - exiting 09:15:00 (4976): No heartbeat from core client for 30 sec - exiting 09:15:01 (4976): No heartbeat from core client for 30 sec - exiting 09:15:02 (4976): No heartbeat from core client for 30 sec - exiting 09:15:03 (4976): No heartbeat from core client for 30 sec - exiting 09:15:04 (4976): No heartbeat from core client for 30 sec - exiting 09:15:05 (4976): No heartbeat from core client for 30 sec - exiting 09:15:07 (4976): No heartbeat from core client for 30 sec - exiting 09:15:08 (4976): No heartbeat from core client for 30 sec - exiting 09:15:09 (4976): No heartbeat from core client for 30 sec - exiting 09:15:10 (4976): No heartbeat from core client for 30 sec - exiting 09:15:11 (4976): No heartbeat from core client for 30 sec - exiting 09:15:12 (4976): No heartbeat from core client for 30 sec - exiting 09:15:13 (4976): No heartbeat from core client for 30 sec - exiting 09:15:14 (4976): No heartbeat from core client for 30 sec - exiting 09:15:15 (4976): No heartbeat from core client for 30 sec - exiting 09:15:16 (4976): No heartbeat from core client for 30 sec - exiting 09:15:17 (4976): No heartbeat from core client for 30 sec - exiting 09:15:18 (4976): No heartbeat from core client for 30 sec - exiting 09:15:19 (4976): No heartbeat from core client for 30 sec - exiting 09:15:20 (4976): No heartbeat from core client for 30 sec - exiting 09:15:21 (4976): No heartbeat from core client for 30 sec - exiting 09:15:22 (4976): No heartbeat from core client for 30 sec - exiting 09:15:23 (4976): No heartbeat from core client for 30 sec - exiting 09:15:24 (4976): No heartbeat from core client for 30 sec - exiting 09:15:25 (4976): No heartbeat from core client for 30 sec - exiting 09:15:26 (4976): No heartbeat from core client for 30 sec - exiting 09:15:27 (4976): No heartbeat from core client for 30 sec - exiting 09:15:28 (4976): No heartbeat from core client for 30 sec - exiting 09:15:29 (4976): No heartbeat from core client for 30 sec - exiting 09:15:30 (4976): No heartbeat from core client for 30 sec - exiting 09:15:31 (4976): No heartbeat from core client for 30 sec - exiting 09:15:32 (4976): No heartbeat from core client for 30 sec - exiting 09:15:33 (4976): No heartbeat from core client for 30 sec - exiting 09:15:34 (4976): No heartbeat from core client for 30 sec - exiting 09:15:35 (4976): No heartbeat from core client for 30 sec - exiting 09:15:36 (4976): No heartbeat from core client for 30 sec - exiting 09:15:37 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5808, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4536, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4480, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4292, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> <file_xfer_error> <file_name>hadam3p_eu_2u33_1976_1_007385233_1_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_2u33_1976_1_007385233_1_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Aug 2011 01:30:05	890420	13195349	hadam3p_eu_2u33_1976_1_007385233_1	115,296	341,802	2.9646
23 Aug 2011 11:09:15	890420	13195349	hadam3p_eu_2u33_1976_1_007385233_1	103,776	307,302	2.9612
22 Aug 2011 10:17:48	890420	13195349	hadam3p_eu_2u33_1976_1_007385233_1	92,256	273,142	2.9607
19 Aug 2011 08:13:01	890420	13195349	hadam3p_eu_2u33_1976_1_007385233_1	80,736	238,779	2.9575
18 Aug 2011 06:49:52	890420	13195349	hadam3p_eu_2u33_1976_1_007385233_1	69,216	204,560	2.9554
17 Aug 2011 06:03:16	890420	13195349	hadam3p_eu_2u33_1976_1_007385233_1	57,696	170,197	2.9499
09 Aug 2011 05:17:59	890420	13195349	hadam3p_eu_2u33_1976_1_007385233_1	46,176	138,073	2.9901
08 Aug 2011 04:15:40	890420	13195349	hadam3p_eu_2u33_1976_1_007385233_1	34,656	103,907	2.9982
05 Aug 2011 03:09:30	890420	13195349	hadam3p_eu_2u33_1976_1_007385233_1	23,136	69,609	3.0087
04 Aug 2011 02:17:16	890420	13195349	hadam3p_eu_2u33_1976_1_007385233_1	11,616	34,903	3.0047