Task 15633607

Name	hadcm3n_3iqh_1980_40_008319307_0
Workunit	8470442
Created	24 Feb 2013, 9:00:05 UTC
Sent	24 Feb 2013, 9:00:11 UTC
Report deadline	26 May 2013, 16:27:22 UTC
Received	1 May 2013, 14:43:17 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1264310
Run time	16 days 13 hours 27 min 1 sec
CPU time	15 days 1 hours 27 min 45 sec
Validate state	Invalid
Credit	10,575.36
Device peak FLOPS	2.93 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5660, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5660, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5660, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5660, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5660, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:29:04 (5272): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 20:29:05 (5272): No heartbeat from core client for 30 sec - exiting 20:29:06 (5272): No heartbeat from core client for 30 sec - exiting 20:29:07 (5272): No heartbeat from core client for 30 sec - exiting 20:29:08 (5272): No heartbeat from core client for 30 sec - exiting 20:29:09 (5272): No heartbeat from core client for 30 sec - exiting 20:29:10 (5272): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7032, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:40:13 (6364): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 15:40:19 (6364): No heartbeat from core client for 30 sec - exiting 15:40:20 (6364): No heartbeat from core client for 30 sec - exiting 15:40:21 (6364): No heartbeat from core client for 30 sec - exiting 15:40:22 (6364): No heartbeat from core client for 30 sec - exiting 15:40:23 (6364): No heartbeat from core client for 30 sec - exiting 15:40:24 (6364): No heartbeat from core client for 30 sec - exiting 15:40:25 (6364): No heartbeat from core client for 30 sec - exiting 15:40:26 (6364): No heartbeat from core client for 30 sec - exiting 15:40:27 (6364): No heartbeat from core client for 30 sec - exiting 15:40:28 (6364): No heartbeat from core client for 30 sec - exiting 15:40:29 (6364): No heartbeat from core client for 30 sec - exiting 15:40:30 (6364): No heartbeat from core client for 30 sec - exiting 15:40:31 (6364): No heartbeat from core client for 30 sec - exiting 15:40:32 (6364): No heartbeat from core client for 30 sec - exiting 15:40:33 (6364): No heartbeat from core client for 30 sec - exiting 15:40:34 (6364): No heartbeat from core client for 30 sec - exiting 15:40:35 (6364): No heartbeat from core client for 30 sec - exiting 15:40:36 (6364): No heartbeat from core client for 30 sec - exiting 15:40:37 (6364): No heartbeat from core client for 30 sec - exiting 15:40:40 (6364): No heartbeat from core client for 30 sec - exiting 15:40:41 (6364): No heartbeat from core client for 30 sec - exiting 15:40:42 (6364): No heartbeat from core client for 30 sec - exiting 15:40:43 (6364): No heartbeat from core client for 30 sec - exiting 15:40:44 (6364): No heartbeat from core client for 30 sec - exiting 15:40:45 (6364): No heartbeat from core client for 30 sec - exiting 15:40:46 (6364): No heartbeat from core client for 30 sec - exiting 15:40:47 (6364): No heartbeat from core client for 30 sec - exiting 15:40:48 (6364): No heartbeat from core client for 30 sec - exiting 15:40:49 (6364): No heartbeat from core client for 30 sec - exiting 15:40:54 (6364): No heartbeat from core client for 30 sec - exiting 15:40:55 (6364): No heartbeat from core client for 30 sec - exiting 15:40:56 (6364): No heartbeat from core client for 30 sec - exiting 15:40:57 (6364): No heartbeat from core client for 30 sec - exiting 15:40:58 (6364): No heartbeat from core client for 30 sec - exiting 15:40:59 (6364): No heartbeat from core client for 30 sec - exiting 15:41:00 (6364): No heartbeat from core client for 30 sec - exiting 15:41:01 (6364): No heartbeat from core client for 30 sec - exiting 15:41:02 (6364): No heartbeat from core client for 30 sec - exiting 15:41:03 (6364): No heartbeat from core client for 30 sec - exiting 15:41:04 (6364): No heartbeat from core client for 30 sec - exiting 15:41:05 (6364): No heartbeat from core client for 30 sec - exiting 15:41:06 (6364): No heartbeat from core client for 30 sec - exiting 15:41:07 (6364): No heartbeat from core client for 30 sec - exiting 15:41:08 (6364): No heartbeat from core client for 30 sec - exiting 15:41:09 (6364): No heartbeat from core client for 30 sec - exiting 15:41:10 (6364): No heartbeat from core client for 30 sec - exiting 15:41:11 (6364): No heartbeat from core client for 30 sec - exiting 15:41:12 (6364): No heartbeat from core client for 30 sec - exiting 15:41:13 (6364): No heartbeat from core client for 30 sec - exiting 15:41:14 (6364): No heartbeat from core client for 30 sec - exiting 15:41:15 (6364): No heartbeat from core client for 30 sec - exiting 15:41:16 (6364): No heartbeat from core client for 30 sec - exiting 15:41:17 (6364): No heartbeat from core client for 30 sec - exiting 15:41:18 (6364): No heartbeat from core client for 30 sec - exiting 15:41:19 (6364): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:49:12 (7024): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 15:49:13 (7024): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:51:42 (9032): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 15:51:57 (9032): No heartbeat from core client for 30 sec - exiting 15:51:58 (9032): No heartbeat from core client for 30 sec - exiting 15:51:59 (9032): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Apr 2013 10:02:15	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	881,280	1,301,227	1.4765
07 Apr 2013 15:45:38	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	855,360	1,261,735	1.4751
06 Apr 2013 10:04:07	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	829,440	1,224,520	1.4763
06 Apr 2013 09:03:45	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	803,520	1,185,017	1.4748
05 Apr 2013 09:04:33	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	777,600	1,147,156	1.4753
04 Apr 2013 09:00:45	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	751,680	1,107,273	1.4731
03 Apr 2013 12:21:58	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	725,760	1,069,055	1.4730
02 Apr 2013 15:55:54	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	699,840	1,033,394	1.4766
02 Apr 2013 09:13:46	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	673,920	1,000,236	1.4842
01 Apr 2013 13:21:06	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	648,000	969,493	1.4961
01 Apr 2013 09:04:29	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	622,080	936,618	1.5056
31 Mar 2013 16:51:15	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	596,160	898,547	1.5072
31 Mar 2013 09:04:12	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	570,240	860,760	1.5095
30 Mar 2013 14:35:25	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	544,320	822,011	1.5102
30 Mar 2013 09:04:27	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	518,400	784,257	1.5128
29 Mar 2013 15:05:34	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	492,480	745,696	1.5142
29 Mar 2013 09:00:59	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	466,560	707,203	1.5158
28 Mar 2013 12:00:19	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	440,640	668,094	1.5162
28 Mar 2013 09:03:45	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	414,720	630,487	1.5203
27 Mar 2013 10:48:04	1264310	15633607	hadcm3n_3iqh_1980_40_008319307_0	388,800	592,727	1.5245