Task 15775540

Name	hadcm3n_37za_1980_40_008365591_0
Workunit	8516450
Created	11 May 2013, 1:36:12 UTC
Sent	11 May 2013, 9:35:05 UTC
Report deadline	10 Aug 2013, 17:02:16 UTC
Received	28 Nov 2013, 0:21:31 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1185918
Run time	8 hours 7 min 59 sec
CPU time	8 hours 7 min 59 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.25 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>5.4.9</core_client_version> <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 20:32:19 (5680): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:42:56 (1080): start_timer_thread(): CreateThread() failed, errno 0 21:35:40 (4168): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... 20:45:23 (4408): start_timer_thread(): CreateThread() failed, errno 0 23:04:08 (5996): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:21:52 (4216): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:13:42 (3440): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:45:44 (4628): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:16:17 (5152): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:41:58 (5640): start_timer_thread(): CreateThread() failed, errno 0 23:12:20 (4208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:12:21 (4208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 17:59:41 (4552): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:31:04 (4136): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 15:53:56 (5980): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 17:25:10 (5432): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:59:27 (4804): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 15:38:20 (3832): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 12:23:16 (5332): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:20:08 (4152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:20:09 (4152): No heartbeat from core client for 30 sec - exiting 20:21:01 (5908): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:31:54 (5484): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:54:58 (5508): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:54:11 (5868): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 13:14:57 (5516): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:05:39 (3344): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 23:31:27 (5084): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:15:17 (2888): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:31:29 (5348): start_timer_thread(): CreateThread() failed, errno 0 20:14:22 (4520): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:53:00 (3764): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 23:27:38 (5592): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 22:08:57 (4596): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:21:48 (5944): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 23:02:01 (4280): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:30:19 (4684): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 18:43:19 (5000): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 22:17:31 (4736): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:02:21 (4412): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:28:37 (5164): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:17:52 (5400): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:15:28 (884): start_timer_thread(): CreateThread() failed, errno 0 21:26:04 (5404): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 17:44:09 (2488): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:55:32 (5312): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5312, selfPID=5312, iMonCtr=1 16:14:18 (4900): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:11:45 (3272): start_timer_thread(): CreateThread() failed, errno 0 21:17:11 (3100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:26:33 (4744): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 12:12:24 (4972): start_timer_thread(): CreateThread() failed, errno 0 13:42:21 (2628): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 18:44:40 (1744): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:48:43 (3636): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 22:37:18 (3508): start_timer_thread(): CreateThread() failed, errno 0 22:49:19 (4272): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:23:47 (196): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 12:25:04 (3632): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:16:05 (1600): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:55:40 (5940): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 22:08:17 (4752): start_timer_thread(): CreateThread() failed, errno 0 22:19:04 (5392): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:09:16 (4776): start_timer_thread(): CreateThread() failed, errno 0 20:35:31 (5196): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 07:54:15 (4636): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 23:11:19 (5836): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 23:57:49 (5052): start_timer_thread(): CreateThread() failed, errno 0 00:13:01 (4956): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:41:17 (1724): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:26:40 (5012): start_timer_thread(): CreateThread() failed, errno 0 19:31:40 (4392): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:24:57 (4848): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 22:44:50 (3336): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 15:38:42 (4836): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:36:13 (4376): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:15:14 (560): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:26:15 (5180): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:03:29 (5356): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 13:23:38 (3416): start_timer_thread(): CreateThread() failed, errno 0 19:05:02 (5624): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 07:47:22 (5144): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 22:41:40 (4200): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 01:23:48 (2548): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:30:03 (6132): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:25:47 (5992): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 23:07:57 (5264): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:50:27 (716): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 17:35:11 (5408): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:28:55 (5424): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:27:44 (4600): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:08:52 (3312): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 14:53:30 (2720): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:33:34 (5060): start_timer_thread(): CreateThread() failed, errno 0 20:57:53 (824): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:07:09 (5788): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:54:05 (4868): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 22:21:05 (6080): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:19:11 (5504): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:33:42 (4252): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 16:39:19 (5100): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 13:19:02 (5664): start_timer_thread(): CreateThread() failed, errno 0 22:11:57 (5944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:23:24 (5892): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:36:19 (4872): start_timer_thread(): CreateThread() failed, errno 0 22:43:23 (452): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:54:47 (6076): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:55:17 (4964): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 23:06:31 (2116): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2116, selfPID=2116, iMonCtr=1 18:16:19 (4784): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 17:05:37 (5648): start_timer_thread(): CreateThread() failed, errno 0 17:17:53 (5616): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:56:05 (6004): start_timer_thread(): CreateThread() failed, errno 0 21:03:45 (5728): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:31:34 (4268): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 22:44:06 (2952): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:35:25 (4516): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:35:09 (4488): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:05:25 (4776): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:22:15 (2676): start_timer_thread(): CreateThread() failed, errno 0 20:28:53 (5244): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:31:51 (4244): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 20:35:25 (2700): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:33:33 (4552): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 18:26:29 (4580): start_timer_thread(): CreateThread() failed, errno 0 18:45:24 (2224): No heartbeat from core client for 30 sec - exiting 18:45:25 (2224): No heartbeat from core client for 30 sec - exiting 18:45:27 (2224): No heartbeat from core client for 30 sec - exiting 18:45:28 (2224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:16:46 (408): start_timer_thread(): CreateThread() failed, errno 0 21:50:37 (4608): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 17:30:26 (5784): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:14:00 (5420): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 22:14:34 (4704): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:06:33 (5028): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:16:51 (3996): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 17:04:06 (5880): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:18:20 (5456): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 23:29:46 (5580): start_timer_thread(): CreateThread() failed, errno 0 Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... 00:18:36 (2188): start_timer_thread(): CreateThread() failed, errno 0 Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... 00:19:06 (2568): start_timer_thread(): CreateThread() failed, errno 0 Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... 00:19:24 (7192): start_timer_thread(): CreateThread() failed, errno 0 Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... 00:19:44 (7720): start_timer_thread(): CreateThread() failed, errno 0 Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... 00:20:15 (5048): start_timer_thread(): CreateThread() failed, errno 0 Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Nov 2013 00:07:53	1185918	15775540	hadcm3n_37za_1980_40_008365591_0	259,200	39,863	0.1538
10 Nov 2013 22:29:08	1185918	15775540	hadcm3n_37za_1980_40_008365591_0	233,280	34,836	0.1493
15 Oct 2013 20:10:35	1185918	15775540	hadcm3n_37za_1980_40_008365591_0	207,360	30,595	0.1475
06 Oct 2013 21:12:19	1185918	15775540	hadcm3n_37za_1980_40_008365591_0	181,440	22,107	0.1218
15 Sep 2013 21:27:41	1185918	15775540	hadcm3n_37za_1980_40_008365591_0	155,520	29,182	0.1876
26 Aug 2013 20:52:02	1185918	15775540	hadcm3n_37za_1980_40_008365591_0	129,600	21,714	0.1675
14 Aug 2013 21:51:04	1185918	15775540	hadcm3n_37za_1980_40_008365591_0	103,680	36,814	0.3551
04 Jul 2013 14:02:34	1185918	15775540	hadcm3n_37za_1980_40_008365591_0	77,760	21,087	0.2712
11 Jun 2013 20:38:47	1185918	15775540	hadcm3n_37za_1980_40_008365591_0	51,840	22,554	0.4351
28 May 2013 18:21:12	1185918	15775540	hadcm3n_37za_1980_40_008365591_0	25,920	30,273	1.1679