Task 15555650

Name	hadcm3n_3dpx_1940_40_008265940_1
Workunit	8421064
Created	23 Jan 2013, 14:18:20 UTC
Sent	23 Jan 2013, 14:18:28 UTC
Report deadline	24 Apr 2013, 21:45:39 UTC
Received	15 Feb 2013, 16:48:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	914050
Run time	5 days 12 hours 43 min 47 sec
CPU time	4 days 17 hours 58 min 5 sec
Validate state	Invalid
Credit	1,555.20
Device peak FLOPS	1.48 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:38:35 (1868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:37:26 (4312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:35:11 (2996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:33:05 (5188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:18:55 (4988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:18:56 (4988): No heartbeat from core client for 30 sec - exiting 01:18:57 (4988): No heartbeat from core client for 30 sec - exiting 02:19:01 (4128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:06:49 (3260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:23:11 (4728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:22:07 (5356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:21:01 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:19:55 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:41:00 (4424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:14:40 (5836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:08:17 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:05:12 (4856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:02:41 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:49:57 (5736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:44:26 (2112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:45:36 (5928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:31 (3156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:34:19 (3576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:34:20 (3576): No heartbeat from core client for 30 sec - exiting 03:35:18 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:43:18 (3672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:40:12 (6136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:39:10 (3556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:38:06 (3228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:41:31 (5308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:37:04 (2068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:15:42 (5792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:14:33 (6012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:12:26 (5132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4992, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 01:00:03 (3244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:53:05 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:54:16 (5748): No heartbeat from core client for 30 sec - exiting 01:54:17 (5748): No heartbeat from core client for 30 sec - exiting 01:54:19 (5748): No heartbeat from core client for 30 sec - exiting 01:54:20 (5748): No heartbeat from core client for 30 sec - exiting 01:54:21 (5748): No heartbeat from core client for 30 sec - exiting 01:54:22 (5748): No heartbeat from core client for 30 sec - exiting 01:54:23 (5748): No heartbeat from core client for 30 sec - exiting 01:55:19 (5748): No heartbeat from core client for 30 sec - exiting 01:55:20 (5748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:55:21 (5748): No heartbeat from core client for 30 sec - exiting 01:55:22 (5748): No heartbeat from core client for 30 sec - exiting 01:55:23 (5748): No heartbeat from core client for 30 sec - exiting 01:55:24 (5748): No heartbeat from core client for 30 sec - exiting 01:55:26 (5748): No heartbeat from core client for 30 sec - exiting 01:55:27 (5748): No heartbeat from core client for 30 sec - exiting 01:55:28 (5748): No heartbeat from core client for 30 sec - exiting 01:55:29 (5748): No heartbeat from core client for 30 sec - exiting 01:55:30 (5748): No heartbeat from core client for 30 sec - exiting 01:55:31 (5748): No heartbeat from core client for 30 sec - exiting 01:57:02 (4884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:15 (1352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:36:46 (4764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:36:47 (4764): No heartbeat from core client for 30 sec - exiting 02:38:03 (4660): No heartbeat from core client for 30 sec - exiting 02:38:04 (4660): No heartbeat from core client for 30 sec - exiting 02:38:05 (4660): No heartbeat from core client for 30 sec - exiting 02:38:06 (4660): No heartbeat from core client for 30 sec - exiting 02:38:07 (4660): No heartbeat from core client for 30 sec - exiting 02:38:09 (4660): No heartbeat from core client for 30 sec - exiting 02:38:10 (4660): No heartbeat from core client for 30 sec - exiting 02:38:11 (4660): No heartbeat from core client for 30 sec - exiting 02:38:12 (4660): No heartbeat from core client for 30 sec - exiting 02:38:13 (4660): No heartbeat from core client for 30 sec - exiting 02:38:15 (4660): No heartbeat from core client for 30 sec - exiting 02:38:16 (4660): No heartbeat from core client for 30 sec - exiting 02:38:17 (4660): No heartbeat from core client for 30 sec - exiting 02:38:18 (4660): No heartbeat from core client for 30 sec - exiting 02:38:19 (4660): No heartbeat from core client for 30 sec - exiting 02:38:21 (4660): No heartbeat from core client for 30 sec - exiting 02:38:22 (4660): No heartbeat from core client for 30 sec - exiting 02:38:23 (4660): No heartbeat from core client for 30 sec - exiting 02:38:24 (4660): No heartbeat from core client for 30 sec - exiting 02:38:25 (4660): No heartbeat from core client for 30 sec - exiting 02:38:26 (4660): No heartbeat from core client for 30 sec - exiting 02:38:27 (4660): No heartbeat from core client for 30 sec - exiting 02:38:28 (4660): No heartbeat from core client for 30 sec - exiting 02:38:29 (4660): No heartbeat from core client for 30 sec - exiting 02:38:31 (4660): No heartbeat from core client for 30 sec - exiting 02:38:32 (4660): No heartbeat from core client for 30 sec - exiting 02:38:33 (4660): No heartbeat from core client for 30 sec - exiting 02:38:34 (4660): No heartbeat from core client for 30 sec - exiting 02:38:35 (4660): No heartbeat from core client for 30 sec - exiting 02:38:36 (4660): No heartbeat from core client for 30 sec - exiting 02:38:37 (4660): No heartbeat from core client for 30 sec - exiting 02:38:39 (4660): No heartbeat from core client for 30 sec - exiting 02:38:40 (4660): No heartbeat from core client for 30 sec - exiting 02:38:41 (4660): No heartbeat from core client for 30 sec - exiting 02:38:42 (4660): No heartbeat from core client for 30 sec - exiting 02:38:43 (4660): No heartbeat from core client for 30 sec - exiting 02:38:44 (4660): No heartbeat from core client for 30 sec - exiting 02:38:45 (4660): No heartbeat from core client for 30 sec - exiting 02:38:46 (4660): No heartbeat from core client for 30 sec - exiting 02:38:48 (4660): No heartbeat from core client for 30 sec - exiting 02:38:49 (4660): No heartbeat from core client for 30 sec - exiting 02:38:50 (4660): No heartbeat from core client for 30 sec - exiting 02:38:51 (4660): No heartbeat from core client for 30 sec - exiting 02:38:52 (4660): No heartbeat from core client for 30 sec - exiting 02:38:53 (4660): No heartbeat from core client for 30 sec - exiting 02:38:54 (4660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:38:56 (4660): No heartbeat from core client for 30 sec - exiting 02:38:57 (4660): No heartbeat from core client for 30 sec - exiting 02:38:58 (4660): No heartbeat from core client for 30 sec - exiting 02:38:59 (4660): No heartbeat from core client for 30 sec - exiting 02:39:00 (4660): No heartbeat from core client for 30 sec - exiting 02:39:01 (4660): No heartbeat from core client for 30 sec - exiting 02:39:02 (4660): No heartbeat from core client for 30 sec - exiting 03:15:15 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:16:01 (368): No heartbeat from core client for 30 sec - exiting 03:16:02 (368): No heartbeat from core client for 30 sec - exiting 03:16:03 (368): No heartbeat from core client for 30 sec - exiting 03:16:04 (368): No heartbeat from core client for 30 sec - exiting 03:16:05 (368): No heartbeat from core client for 30 sec - exiting 03:16:07 (368): No heartbeat from core client for 30 sec - exiting 03:16:08 (368): No heartbeat from core client for 30 sec - exiting 03:16:09 (368): No heartbeat from core client for 30 sec - exiting 03:16:10 (368): No heartbeat from core client for 30 sec - exiting 03:16:11 (368): No heartbeat from core client for 30 sec - exiting 03:16:12 (368): No heartbeat from core client for 30 sec - exiting 03:16:13 (368): No heartbeat from core client for 30 sec - exiting 03:16:14 (368): No heartbeat from core client for 30 sec - exiting 03:16:15 (368): No heartbeat from core client for 30 sec - exiting 03:16:16 (368): No heartbeat from core client for 30 sec - exiting 03:16:18 (368): No heartbeat from core client for 30 sec - exiting 03:16:19 (368): No heartbeat from core client for 30 sec - exiting 03:16:20 (368): No heartbeat from core client for 30 sec - exiting 03:16:21 (368): No heartbeat from core client for 30 sec - exiting 03:16:22 (368): No heartbeat from core client for 30 sec - exiting 03:16:23 (368): No heartbeat from core client for 30 sec - exiting 03:16:24 (368): No heartbeat from core client for 30 sec - exiting 03:16:25 (368): No heartbeat from core client for 30 sec - exiting 03:16:26 (368): No heartbeat from core client for 30 sec - exiting 03:16:27 (368): No heartbeat from core client for 30 sec - exiting 03:16:28 (368): No heartbeat from core client for 30 sec - exiting 03:16:30 (368): No heartbeat from core client for 30 sec - exiting 03:16:31 (368): No heartbeat from core client for 30 sec - exiting 03:16:32 (368): No heartbeat from core client for 30 sec - exiting 03:16:33 (368): No heartbeat from core client for 30 sec - exiting 03:16:34 (368): No heartbeat from core client for 30 sec - exiting 03:16:35 (368): No heartbeat from core client for 30 sec - exiting 03:16:36 (368): No heartbeat from core client for 30 sec - exiting 03:16:37 (368): No heartbeat from core client for 30 sec - exiting 03:16:38 (368): No heartbeat from core client for 30 sec - exiting 03:16:39 (368): No heartbeat from core client for 30 sec - exiting 03:16:41 (368): No heartbeat from core client for 30 sec - exiting 03:16:42 (368): No heartbeat from core client for 30 sec - exiting 03:16:43 (368): No heartbeat from core client for 30 sec - exiting 03:16:44 (368): No heartbeat from core client for 30 sec - exiting 03:16:45 (368): No heartbeat from core client for 30 sec - exiting 03:16:46 (368): No heartbeat from core client for 30 sec - exiting 03:16:47 (368): No heartbeat from core client for 30 sec - exiting 03:16:48 (368): No heartbeat from core client for 30 sec - exiting 03:16:49 (368): No heartbeat from core client for 30 sec - exiting 03:16:50 (368): No heartbeat from core client for 30 sec - exiting 03:16:51 (368): No heartbeat from core client for 30 sec - exiting 03:16:53 (368): No heartbeat from core client for 30 sec - exiting 03:16:54 (368): No heartbeat from core client for 30 sec - exiting 03:16:55 (368): No heartbeat from core client for 30 sec - exiting 03:16:56 (368): No heartbeat from core client for 30 sec - exiting 03:16:57 (368): No heartbeat from core client for 30 sec - exiting 03:16:58 (368): No heartbeat from core client for 30 sec - exiting 03:16:59 (368): No heartbeat from core client for 30 sec - exiting 03:17:00 (368): No heartbeat from core client for 30 sec - exiting 03:17:01 (368): No heartbeat from core client for 30 sec - exiting 03:17:02 (368): No heartbeat from core client for 30 sec - exiting 03:17:03 (368): No heartbeat from core client for 30 sec - exiting 03:17:05 (368): No heartbeat from core client for 30 sec - exiting 03:17:06 (368): No heartbeat from core client for 30 sec - exiting 03:17:07 (368): No heartbeat from core client for 30 sec - exiting 03:17:08 (368): No heartbeat from core client for 30 sec - exiting 03:17:09 (368): No heartbeat from core client for 30 sec - exiting 03:17:10 (368): No heartbeat from core client for 30 sec - exiting 03:17:11 (368): No heartbeat from core client for 30 sec - exiting 03:17:12 (368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:17:13 (368): No heartbeat from core client for 30 sec - exiting 03:39:59 (5788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:01:36 (5496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:39 (4696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5764, iMonCtr=1 Model crash detected, will try to restart... 11:31:29 (5764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5832, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Feb 2013 14:55:39	914050	15555650	hadcm3n_3dpx_1940_40_008265940_1	129,600	380,013	2.9322
08 Feb 2013 07:01:25	914050	15555650	hadcm3n_3dpx_1940_40_008265940_1	103,680	303,802	2.9302
03 Feb 2013 21:19:11	914050	15555650	hadcm3n_3dpx_1940_40_008265940_1	77,760	228,094	2.9333
30 Jan 2013 16:30:39	914050	15555650	hadcm3n_3dpx_1940_40_008265940_1	51,840	151,754	2.9274
27 Jan 2013 00:49:29	914050	15555650	hadcm3n_3dpx_1940_40_008265940_1	25,920	75,537	2.9142