Task 13093289

Name	hadcm3n_y95a_1900_40_007344712_0
Workunit	7542142
Created	6 Jul 2011, 13:26:22 UTC
Sent	22 Jul 2011, 13:34:24 UTC
Report deadline	21 Oct 2011, 21:01:35 UTC
Received	21 Aug 2011, 22:59:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1157799
Run time	15 days 6 hours 50 min 16 sec
CPU time	15 days 1 hours 0 min 59 sec
Validate state	Invalid
Credit	8,398.08
Device peak FLOPS	2.92 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.26</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 01:52:25 (4164): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:49:00 (1620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:49:01 (1620): No heartbeat from core client for 30 sec - exiting 01:49:02 (1620): No heartbeat from core client for 30 sec - exiting 01:49:03 (1620): No heartbeat from core client for 30 sec - exiting 01:49:04 (1620): No heartbeat from core client for 30 sec - exiting 01:49:05 (1620): No heartbeat from core client for 30 sec - exiting 01:49:06 (1620): No heartbeat from core client for 30 sec - exiting 01:49:07 (1620): No heartbeat from core client for 30 sec - exiting 01:49:09 (1620): No heartbeat from core client for 30 sec - exiting 01:49:10 (1620): No heartbeat from core client for 30 sec - exiting 01:49:11 (1620): No heartbeat from core client for 30 sec - exiting 01:57:43 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:36:40 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:36:41 (3828): No heartbeat from core client for 30 sec - exiting 01:36:42 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 00:37:56 (5016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:43:21 (6876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:13:14 (6204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:01 (4224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:50:07 (4836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5500, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:11:54 (3648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:02:23 (4912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:24:50 (2172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:29:59 (2620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:57:27 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:58:24 (4580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:58:25 (4580): No heartbeat from core client for 30 sec - exiting 14:58:27 (4580): No heartbeat from core client for 30 sec - exiting 14:58:28 (4580): No heartbeat from core client for 30 sec - exiting 14:58:29 (4580): No heartbeat from core client for 30 sec - exiting 14:58:30 (4580): No heartbeat from core client for 30 sec - exiting 14:58:31 (4580): No heartbeat from core client for 30 sec - exiting 14:58:32 (4580): No heartbeat from core client for 30 sec - exiting 14:58:33 (4580): No heartbeat from core client for 30 sec - exiting 14:58:34 (4580): No heartbeat from core client for 30 sec - exiting 14:58:35 (4580): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 15:05:08 (2308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:05:09 (2308): No heartbeat from core client for 30 sec - exiting 15:05:10 (2308): No heartbeat from core client for 30 sec - exiting 15:05:11 (2308): No heartbeat from core client for 30 sec - exiting 15:05:12 (2308): No heartbeat from core client for 30 sec - exiting 15:05:13 (2308): No heartbeat from core client for 30 sec - exiting 15:05:15 (2308): No heartbeat from core client for 30 sec - exiting 15:05:16 (2308): No heartbeat from core client for 30 sec - exiting 15:05:17 (2308): No heartbeat from core client for 30 sec - exiting 15:05:18 (2308): No heartbeat from core client for 30 sec - exiting 15:05:19 (2308): No heartbeat from core client for 30 sec - exiting 15:11:15 (6040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4884, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4884, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4884, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=996, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=996, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=996, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Aug 2011 21:13:22	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	699,840	1,288,364	1.8409
21 Aug 2011 07:35:52	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	673,920	1,242,673	1.8439
21 Aug 2011 07:35:52	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	648,000	1,197,065	1.8473
20 Aug 2011 04:12:58	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	622,080	1,152,096	1.8520
19 Aug 2011 18:47:44	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	596,160	1,107,457	1.8577
19 Aug 2011 13:30:19	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	570,240	1,061,796	1.8620
18 Aug 2011 02:28:19	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	544,320	1,014,269	1.8634
17 Aug 2011 14:33:19	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	518,400	967,531	1.8664
17 Aug 2011 07:52:06	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	492,480	918,551	1.8652
16 Aug 2011 08:08:26	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	466,560	869,068	1.8627
16 Aug 2011 08:08:26	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	440,640	819,789	1.8605
15 Aug 2011 06:53:21	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	414,720	770,299	1.8574
15 Aug 2011 06:53:21	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	388,800	720,724	1.8537
15 Aug 2011 06:53:21	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	362,880	671,060	1.8493
13 Aug 2011 06:21:17	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	336,960	623,019	1.8489
13 Aug 2011 06:21:17	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	311,040	574,840	1.8481
12 Aug 2011 06:48:06	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	285,120	528,813	1.8547
12 Aug 2011 06:48:06	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	259,200	484,466	1.8691
11 Aug 2011 01:10:25	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	233,280	438,794	1.8810
11 Aug 2011 01:10:25	1157799	13093289	hadcm3n_y95a_1900_40_007344712_0	207,360	393,297	1.8967