Task 12660393

Name	hadam3p_saf_1cps_1975_1_006944312_1
Workunit	7147628
Created	11 Mar 2011, 11:35:42 UTC
Sent	11 Mar 2011, 11:52:10 UTC
Report deadline	21 Feb 2012, 17:12:10 UTC
Received	20 Apr 2011, 17:29:34 UTC
Server state	Over
Outcome	No reply
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1070354
Run time	3 days 21 hours 5 min 54 sec
CPU time	3 days 5 hours 57 min 4 sec
Validate state	Invalid
Credit	2,057.21
Device peak FLOPS	2.86 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Southern Africa v6.08 windows_intelx86
Stderr	<core_client_version>6.12.6</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7928, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3980, iMonCtr=2 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9856, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... GloCPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7692, selfPID=76Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5888, selfPID=6256, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... CPDN Monitor - Quit request from BOINC... 11:41:30 (4900): No heartbeat from core client for 30 sec - exiting 11:42:23 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:21:03 (1576): Can't set up shared mem: -1. Will run in standalone mode. No Process Handle Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=1 16:20:28 (1848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=508, iMonCtr=2 CPDN Monitor - Quit request from BOINC... ReGCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... ReCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6504, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3500, selfPID=3500, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5684, selfPID=5684, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:15:37 (6008): No heartbeat from core client for 30 sec - exiting 14:15:38 (6008): No heartbeat from core client for 30 sec - exiting 14:15:39 (6008): No heartbeat from core client for 30 sec - exiting 14:15:40 (6008): No heartbeat from core client for 30 sec - exiting 14:15:41 (6008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4404, selfPID=4404, iMonCtr=2 CPDN Monitor - Quit request from BOINC... 22:28:25 (5408): No heartbeat from core client for 30 sec - exiting 22:28:26 (5408): No heartbeat from core client for 30 sec - exiting 22:28:27 (5408): No heartbeat from core client for 30 sec - exiting 22:28:28 (5408): No heartbeat from core client for 30 sec - exiting 22:28:29 (5408): No heartbeat from core client for 30 sec - exiting 22:28:30 (5408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3908, selfPID=1556, iMonCtr=1 Model crash detected, will try to restart... 09:39:09 (872): No heartbeat from core client for 30 sec - exiting 09:39:10 (872): No heartbeat from core client for 30 sec - exiting 09:39:11 (872): No heartbeat from core client for 30 sec - exiting 09:39:12 (872): No heartbeat from core client for 30 sec - exiting 09:39:13 (872): No heartbeat from core client for 30 sec - exiting 09:39:14 (872): No heartbeat from core client for 30 sec - exiting 09:39:15 (872): No heartbeat from core client for 30 sec - exiting 09:39:16 (872): No heartbeat from core client for 30 sec - exiting 09:39:17 (872): No heartbeat from core client for 30 sec - exiting 09:39:18 (872): No heartbeat from core client for 30 sec - exiting 09:39:19 (872): No heartbeat from core client for 30 sec - exiting 09:39:20 (872): No heartbeat from core client for 30 sec - exiting 09:39:21 (872): No heartbeat from core client for 30 sec - exiting 09:39:22 (872): No heartbeat from core client for 30 sec - exiting 09:39:23 (872): No heartbeat from core client for 30 sec - exiting 09:39:24 (872): No heartbeat from core client for 30 sec - exiting 09:39:25 (872): No heartbeat from core client for 30 sec - exiting 09:39:26 (872): No heartbeat from core client for 30 sec - exiting 09:39:27 (872): No heartbeat from core client for 30 sec - exiting 09:39:28 (872): No heartbeat from core client for 30 sec - exiting 09:39:29 (872): No heartbeat from core client for 30 sec - exiting 09:39:30 (872): No heartbeat from core client for 30 sec - exiting 09:39:31 (872): No heartbeat from core client for 30 sec - exiting 09:39:32 (872): No heartbeat from core client for 30 sec - exiting 09:39:33 (872): No heartbeat from core client for 30 sec - exiting 09:39:34 (872): No heartbeat from core client for 30 sec - exiting 09:39:35 (872): No heartbeat from core client for 30 sec - exiting 09:39:36 (872): No heartbeat from core client for 30 sec - exiting 09:39:37 (872): No heartbeat from core client for 30 sec - exiting 09:39:38 (872): No heartbeat from core client for 30 sec - exiting 09:39:39 (872): No heartbeat from core client for 30 sec - exiting 09:39:40 (872): No heartbeat from core client for 30 sec - exiting 09:39:41 (872): No heartbeat from core client for 30 sec - exiting 09:39:42 (872): No heartbeat from core client for 30 sec - exiting 09:39:43 (872): No heartbeat from core client for 30 sec - exiting 09:39:44 (872): No heartbeat from core client for 30 sec - exiting 09:39:45 (872): No heartbeat from core client for 30 sec - exiting 09:39:46 (872): No heartbeat from core client for 30 sec - exiting 09:39:47 (872): No heartbeat from core client for 30 sec - exiting 09:39:48 (872): No heartbeat from core client for 30 sec - exiting 09:39:49 (872): No heartbeat from core client for 30 sec - exiting 09:39:50 (872): No heartbeat from core client for 30 sec - exiting 09:39:51 (872): No heartbeat from core client for 30 sec - exiting 09:39:52 (872): No heartbeat from core client for 30 sec - exiting 09:39:53 (872): No heartbeat from core client for 30 sec - exiting 09:39:54 (872): No heartbeat from core client for 30 sec - exiting 09:39:55 (872): No heartbeat from core client for 30 sec - exiting 09:39:56 (872): No heartbeat from core client for 30 sec - exiting 09:39:57 (872): No heartbeat from core client for 30 sec - exiting 09:39:58 (872): No heartbeat from core client for 30 sec - exiting 09:39:59 (872): No heartbeat from core client for 30 sec - exiting 09:40:00 (872): No heartbeat from core client for 30 sec - exiting 09:40:01 (872): No heartbeat from core client for 30 sec - exiting 09:40:33 (872): No heartbeat from core client for 30 sec - exiting 09:40:34 (872): No heartbeat from core client for 30 sec - exiting 09:40:35 (872): No heartbeat from core client for 30 sec - exiting 09:40:36 (872): No heartbeat from core client for 30 sec - exiting 09:40:37 (872): No heartbeat from core client for 30 sec - exiting 09:40:38 (872): No heartbeat from core client for 30 sec - exiting 09:40:39 (872): No heartbeat from core client for 30 sec - exiting 09:40:40 (872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:44:10 (5164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:45 (4552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6044, selfPID=6044, iMonCtr=2 10:05:53 (5448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2764, selfPID=2764, iMonCtr=2 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3268, iMonCtr=2 17:21:29 (3756): No heartbeat from core client for 30 sec - exiting 17:21:31 (3756): No heartbeat from core client for 30 sec - exiting 17:21:32 (3756): No heartbeat from core client for 30 sec - exiting 17:21:33 (3756): No heartbeat from core client for 30 sec - exiting 17:21:34 (3756): No heartbeat from core client for 30 sec - exiting 17:21:35 (3756): No heartbeat from core client for 30 sec - exiting 17:21:36 (3756): No heartbeat from core client for 30 sec - exiting 17:21:37 (3756): No heartbeat from core client for 30 sec - exiting 17:21:38 (3756): No heartbeat from core client for 30 sec - exiting 17:21:39 (3756): No heartbeat from core client for 30 sec - exiting 17:21:40 (3756): No heartbeat from core client for 30 sec - exiting 17:21:41 (3756): No heartbeat from core client for 30 sec - exiting 17:21:42 (3756): No heartbeat from core client for 30 sec - exiting 17:21:43 (3756): No heartbeat from core client for 30 sec - exiting 17:21:44 (3756): No heartbeat from core client for 30 sec - exiting 17:21:45 (3756): No heartbeat from core client for 30 sec - exiting 17:21:46 (3756): No heartbeat from core client for 30 sec - exiting 17:21:47 (3756): No heartbeat from core client for 30 sec - exiting 17:21:48 (3756): No heartbeat from core client for 30 sec - exiting 17:21:49 (3756): No heartbeat from core client for 30 sec - exiting 17:21:50 (3756): No heartbeat from core client for 30 sec - exiting 17:21:51 (3756): No heartbeat from core client for 30 sec - exiting 17:21:52 (3756): No heartbeat from core client for 30 sec - exiting 17:21:53 (3756): No heartbeat from core client for 30 sec - exiting 17:21:54 (3756): No heartbeat from core client for 30 sec - exiting 17:21:55 (3756): No heartbeat from core client for 30 sec - exiting 17:21:56 (3756): No heartbeat from core client for 30 sec - exiting 17:21:57 (3756): No heartbeat from core client for 30 sec - exiting 17:21:58 (3756): No heartbeat from core client for 30 sec - exiting 17:21:59 (3756): No heartbeat from core client for 30 sec - exiting 17:22:00 (3756): No heartbeat from core client for 30 sec - exiting 17:22:01 (3756): No heartbeat from core client for 30 sec - exiting 17:22:02 (3756): No heartbeat from core client for 30 sec - exiting 17:22:03 (3756): No heartbeat from core client for 30 sec - exiting 17:22:04 (3756): No heartbeat from core client for 30 sec - exiting 17:22:05 (3756): No heartbeat from core client for 30 sec - exiting 17:22:06 (3756): No heartbeat from core client for 30 sec - exiting 17:22:07 (3756): No heartbeat from core client for 30 sec - exiting 17:22:08 (3756): No heartbeat from core client for 30 sec - exiting 17:22:09 (3756): No heartbeat from core client for 30 sec - exiting 17:22:10 (3756): No heartbeat from core client for 30 sec - exiting 17:22:11 (3756): No heartbeat from core client for 30 sec - exiting 17:22:12 (3756): No heartbeat from core client for 30 sec - exiting 17:22:13 (3756): No heartbeat from core client for 30 sec - exiting 17:22:14 (3756): No heartbeat from core client for 30 sec - exiting 17:22:15 (3756): No heartbeat from core client for 30 sec - exiting 17:22:16 (3756): No heartbeat from core client for 30 sec - exiting 17:22:17 (3756): No heartbeat from core client for 30 sec - exiting 17:22:18 (3756): No heartbeat from core client for 30 sec - exiting 17:22:19 (3756): No heartbeat from core client for 30 sec - exiting 17:22:20 (3756): No heartbeat from core client for 30 sec - exiting 17:22:51 (3756): No heartbeat from core client for 30 sec - exiting 17:22:52 (3756): No heartbeat from core client for 30 sec - exiting 17:22:53 (3756): No heartbeat from core client for 30 sec - exiting 17:22:54 (3756): No heartbeat from core client for 30 sec - exiting 17:22:55 (3756): No heartbeat from core client for 30 sec - exiting 17:22:56 (3756): No heartbeat from core client for 30 sec - exiting 17:22:57 (3756): No heartbeat from core client for 30 sec - exiting 17:22:58 (3756): No heartbeat from core client for 30 sec - exiting 17:22:59 (3756): No heartbeat from core client for 30 sec - exiting 17:23:00 (3756): No heartbeat from core client for 30 sec - exiting 17:23:01 (3756): No heartbeat from core client for 30 sec - exiting 17:23:02 (3756): No heartbeat from core client for 30 sec - exiting 17:23:03 (3756): No heartbeat from core client for 30 sec - exiting 17:23:04 (3756): No heartbeat from core client for 30 sec - exiting 17:23:05 (3756): No heartbeat from core client for 30 sec - exiting 17:23:06 (3756): No heartbeat from core client for 30 sec - exiting 17:23:07 (3756): No heartbeat from core client for 30 sec - exiting 17:23:08 (3756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:32:17 (2264): No heartbeat from core client for 30 sec - exiting 07:32:18 (2264): No heartbeat from core client for 30 sec - exiting 07:32:19 (2264): No heartbeat from core client for 30 sec - exiting 07:32:20 (2264): No heartbeat from core client for 30 sec - exiting 07:32:22 (2264): No heartbeat from core client for 30 sec - exiting 07:32:23 (2264): No heartbeat from core client for 30 sec - exiting 07:32:24 (2264): No heartbeat from core client for 30 sec - exiting 07:32:25 (2264): No heartbeat from core client for 30 sec - exiting 07:32:26 (2264): No heartbeat from core client for 30 sec - exiting 07:32:27 (2264): No heartbeat from core client for 30 sec - exiting 07:32:28 (2264): No heartbeat from core client for 30 sec - exiting 07:32:29 (2264): No heartbeat from core client for 30 sec - exiting 07:32:30 (2264): No heartbeat from core client for 30 sec - exiting 07:32:31 (2264): No heartbeat from core client for 30 sec - exiting 07:32:32 (2264): No heartbeat from core client for 30 sec - exiting 07:32:33 (2264): No heartbeat from core client for 30 sec - exiting 07:32:34 (2264): No heartbeat from core client for 30 sec - exiting 07:32:35 (2264): No heartbeat from core client for 30 sec - exiting 07:32:36 (2264): No heartbeat from core client for 30 sec - exiting 07:32:37 (2264): No heartbeat from core client for 30 sec - exiting 07:32:38 (2264): No heartbeat from core client for 30 sec - exiting 07:32:39 (2264): No heartbeat from core client for 30 sec - exiting 07:32:40 (2264): No heartbeat from core client for 30 sec - exiting 07:32:41 (2264): No heartbeat from core client for 30 sec - exiting 07:32:42 (2264): No heartbeat from core client for 30 sec - exiting 07:32:43 (2264): No heartbeat from core client for 30 sec - exiting 07:32:44 (2264): No heartbeat from core client for 30 sec - exiting 07:32:45 (2264): No heartbeat from core client for 30 sec - exiting 07:32:46 (2264): No heartbeat from core client for 30 sec - exiting 07:32:47 (2264): No heartbeat from core client for 30 sec - exiting 07:32:48 (2264): No heartbeat from core client for 30 sec - exiting 07:32:49 (2264): No heartbeat from core client for 30 sec - exiting 07:32:50 (2264): No heartbeat from core client for 30 sec - exiting 07:32:51 (2264): No heartbeat from core client for 30 sec - exiting 07:32:52 (2264): No heartbeat from core client for 30 sec - exiting 07:32:53 (2264): No heartbeat from core client for 30 sec - exiting 07:32:54 (2264): No heartbeat from core client for 30 sec - exiting 07:32:55 (2264): No heartbeat from core client for 30 sec - exiting 07:32:57 (2264): No heartbeat from core client for 30 sec - exiting 07:32:58 (2264): No heartbeat from core client for 30 sec - exiting 07:32:59 (2264): No heartbeat from core client for 30 sec - exiting 07:33:00 (2264): No heartbeat from core client for 30 sec - exiting 07:33:01 (2264): No heartbeat from core client for 30 sec - exiting 07:33:02 (2264): No heartbeat from core client for 30 sec - exiting 07:33:03 (2264): No heartbeat from core client for 30 sec - exiting 07:33:04 (2264): No heartbeat from core client for 30 sec - exiting 07:33:05 (2264): No heartbeat from core client for 30 sec - exiting 07:33:06 (2264): No heartbeat from core client for 30 sec - exiting 07:33:07 (2264): No heartbeat from core client for 30 sec - exiting 07:33:08 (2264): No heartbeat from core client for 30 sec - exiting 07:33:09 (2264): No heartbeat from core client for 30 sec - exiting 07:33:10 (2264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:16 (6016): No heartbeat from core client for 30 sec - exiting 10:48:17 (6016): No heartbeat from core client for 30 sec - exiting 10:48:18 (6016): No heartbeat from core client for 30 sec - exiting 10:48:19 (6016): No heartbeat from core client for 30 sec - exiting 10:48:20 (6016): No heartbeat from core client for 30 sec - exiting 10:48:22 (6016): No heartbeat from core client for 30 sec - exiting 10:48:23 (6016): No heartbeat from core client for 30 sec - exiting 10:48:24 (6016): No heartbeat from core client for 30 sec - exiting 10:48:25 (6016): No heartbeat from core client for 30 sec - exiting 10:48:26 (6016): No heartbeat from core client for 30 sec - exiting 10:48:27 (6016): No heartbeat from core client for 30 sec - exiting 10:48:28 (6016): No heartbeat from core client for 30 sec - exiting 10:48:29 (6016): No heartbeat from core client for 30 sec - exiting 10:48:30 (6016): No heartbeat from core client for 30 sec - exiting 10:48:31 (6016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4744, selfPID=4744, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6064, selfPID=6064, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3500, selfPID=3500, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7516, selfPID=7516, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7612, selfPID=7612, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Apr 2011 19:07:04	1070354	12660393	hadam3p_saf_1cps_1975_1_006944312_1	126,816	270,718	2.1347
20 Apr 2011 19:07:04	1070354	12660393	hadam3p_saf_1cps_1975_1_006944312_1	115,296	247,522	2.1468
20 Apr 2011 19:07:04	1070354	12660393	hadam3p_saf_1cps_1975_1_006944312_1	103,776	221,923	2.1385
20 Apr 2011 19:07:04	1070354	12660393	hadam3p_saf_1cps_1975_1_006944312_1	92,256	196,948	2.1348
10 Apr 2011 12:22:40	1070354	12660393	hadam3p_saf_1cps_1975_1_006944312_1	80,736	173,506	2.1491
08 Apr 2011 06:11:51	1070354	12660393	hadam3p_saf_1cps_1975_1_006944312_1	69,216	150,445	2.1736
06 Apr 2011 07:11:25	1070354	12660393	hadam3p_saf_1cps_1975_1_006944312_1	57,696	126,873	2.1990
03 Apr 2011 17:05:00	1070354	12660393	hadam3p_saf_1cps_1975_1_006944312_1	46,176	102,475	2.2192
02 Apr 2011 18:37:40	1070354	12660393	hadam3p_saf_1cps_1975_1_006944312_1	34,656	75,754	2.1859
01 Apr 2011 09:40:31	1070354	12660393	hadam3p_saf_1cps_1975_1_006944312_1	23,136	53,470	2.3111
14 Mar 2011 18:05:45	1070354	12660393	hadam3p_saf_1cps_1975_1_006944312_1	11,616	26,119	2.2485