Task 13131938

Name	hadcm3n_yl11_1900_40_007360111_2
Workunit	7557541
Created	7 Jul 2011, 21:13:56 UTC
Sent	7 Jul 2011, 21:16:01 UTC
Report deadline	7 Oct 2011, 4:43:12 UTC
Received	4 Aug 2011, 9:16:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1195625
Run time	14 days 7 hours 22 min 58 sec
CPU time	12 days 3 hours 10 min 36 sec
Validate state	Invalid
Credit	9,953.28
Device peak FLOPS	3.77 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5312, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:49:49 (2240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:49:50 (2240): No heartbeat from core client for 30 sec - exiting 11:49:51 (2240): No heartbeat from core client for 30 sec - exiting 11:49:52 (2240): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4168, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 05:01:57 (5268): No heartbeat from core client for 30 sec - exiting 05:01:58 (5268): No heartbeat from core client for 30 sec - exiting 05:01:59 (5268): No heartbeat from core client for 30 sec - exiting 05:02:00 (5268): No heartbeat from core client for 30 sec - exiting 05:02:01 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:53:56 (4720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=620, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4904, iMonCtr=1 Model crash detected, will try to restart... 21:17:55 (4792): No heartbeat from core client for 30 sec - exiting 21:17:56 (4792): No heartbeat from core client for 30 sec - exiting 21:17:57 (4792): No heartbeat from core client for 30 sec - exiting 21:17:58 (4792): No heartbeat from core client for 30 sec - exiting 21:17:59 (4792): No heartbeat from core client for 30 sec - exiting 21:18:00 (4792): No heartbeat from core client for 30 sec - exiting 21:18:01 (4792): No heartbeat from core client for 30 sec - exiting 21:18:02 (4792): No heartbeat from core client for 30 sec - exiting 21:18:03 (4792): No heartbeat from core client for 30 sec - exiting 21:18:04 (4792): No heartbeat from core client for 30 sec - exiting 21:18:05 (4792): No heartbeat from core client for 30 sec - exiting 21:18:06 (4792): No heartbeat from core client for 30 sec - exiting 21:18:07 (4792): No heartbeat from core client for 30 sec - exiting 21:18:08 (4792): No heartbeat from core client for 30 sec - exiting 21:18:09 (4792): No heartbeat from core client for 30 sec - exiting 21:18:10 (4792): No heartbeat from core client for 30 sec - exiting 21:18:11 (4792): No heartbeat from core client for 30 sec - exiting 21:18:12 (4792): No heartbeat from core client for 30 sec - exiting 21:18:13 (4792): No heartbeat from core client for 30 sec - exiting 21:18:14 (4792): No heartbeat from core client for 30 sec - exiting 21:18:15 (4792): No heartbeat from core client for 30 sec - exiting 21:18:16 (4792): No heartbeat from core client for 30 sec - exiting 21:18:17 (4792): No heartbeat from core client for 30 sec - exiting 21:18:18 (4792): No heartbeat from core client for 30 sec - exiting 21:18:19 (4792): No heartbeat from core client for 30 sec - exiting 21:18:20 (4792): No heartbeat from core client for 30 sec - exiting 21:18:21 (4792): No heartbeat from core client for 30 sec - exiting 21:18:22 (4792): No heartbeat from core client for 30 sec - exiting 21:18:23 (4792): No heartbeat from core client for 30 sec - exiting 21:18:24 (4792): No heartbeat from core client for 30 sec - exiting 21:18:25 (4792): No heartbeat from core client for 30 sec - exiting 21:18:26 (4792): No heartbeat from core client for 30 sec - exiting 21:18:27 (4792): No heartbeat from core client for 30 sec - exiting 21:18:28 (4792): No heartbeat from core client for 30 sec - exiting 21:18:29 (4792): No heartbeat from core client for 30 sec - exiting 21:18:30 (4792): No heartbeat from core client for 30 sec - exiting 21:18:31 (4792): No heartbeat from core client for 30 sec - exiting 21:18:32 (4792): No heartbeat from core client for 30 sec - exiting 21:18:33 (4792): No heartbeat from core client for 30 sec - exiting 21:18:34 (4792): No heartbeat from core client for 30 sec - exiting 21:18:35 (4792): No heartbeat from core client for 30 sec - exiting 21:18:36 (4792): No heartbeat from core client for 30 sec - exiting 21:18:37 (4792): No heartbeat from core client for 30 sec - exiting 21:18:38 (4792): No heartbeat from core client for 30 sec - exiting 21:18:39 (4792): No heartbeat from core client for 30 sec - exiting 21:18:40 (4792): No heartbeat from core client for 30 sec - exiting 21:18:41 (4792): No heartbeat from core client for 30 sec - exiting 21:18:42 (4792): No heartbeat from core client for 30 sec - exiting 21:18:43 (4792): No heartbeat from core client for 30 sec - exiting 21:18:44 (4792): No heartbeat from core client for 30 sec - exiting 21:18:45 (4792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4052, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4052, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4052, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4052, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4052, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4052, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Aug 2011 04:01:23	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	829,440	1,033,962	1.2466
03 Aug 2011 18:07:08	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	803,520	1,001,622	1.2465
03 Aug 2011 08:47:49	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	777,600	969,442	1.2467
02 Aug 2011 23:03:14	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	751,680	937,418	1.2471
02 Aug 2011 13:04:52	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	725,760	905,252	1.2473
02 Aug 2011 00:19:08	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	699,840	872,761	1.2471
01 Aug 2011 13:55:32	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	673,920	840,165	1.2467
01 Aug 2011 03:40:20	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	648,000	807,698	1.2464
31 Jul 2011 17:47:38	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	622,080	775,502	1.2466
31 Jul 2011 07:05:40	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	596,160	743,442	1.2471
30 Jul 2011 20:30:43	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	570,240	711,432	1.2476
30 Jul 2011 09:23:56	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	544,320	679,385	1.2481
29 Jul 2011 21:42:53	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	518,400	647,023	1.2481
29 Jul 2011 09:45:00	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	492,480	615,190	1.2492
28 Jul 2011 16:32:45	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	466,560	583,033	1.2496
28 Jul 2011 05:22:14	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	440,640	550,226	1.2487
27 Jul 2011 17:23:07	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	414,720	518,061	1.2492
27 Jul 2011 06:27:33	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	388,800	485,415	1.2485
26 Jul 2011 19:51:50	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	362,880	452,988	1.2483
26 Jul 2011 09:28:21	1153877	13131938	hadcm3n_yl11_1900_40_007360111_2	336,960	420,477	1.2479