Task 12991861

Name	hadcm3n_s6fl_1940_40_007300538_0
Workunit	7497962
Created	20 Jun 2011, 23:51:45 UTC
Sent	20 Jun 2011, 23:52:54 UTC
Report deadline	20 Sep 2011, 7:20:05 UTC
Received	7 Jul 2011, 10:24:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1108969
Run time	8 days 10 hours 13 min
CPU time	6 days 3 hours 16 min 14 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.04 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6396, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:00:36 (4696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:14:18 (4272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:20:20 (8996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:53:59 (5176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:20:44 (2064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:44:05 (10668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:26:26 (6492): No heartbeat from core client for 30 sec - exiting 22:26:43 (6492): No heartbeat from core client for 30 sec - exiting 22:26:44 (6492): No heartbeat from core client for 30 sec - exiting 22:26:45 (6492): No heartbeat from core client for 30 sec - exiting 22:26:46 (6492): No heartbeat from core client for 30 sec - exiting 22:26:47 (6492): No heartbeat from core client for 30 sec - exiting 22:26:48 (6492): No heartbeat from core client for 30 sec - exiting 22:26:49 (6492): No heartbeat from core client for 30 sec - exiting 22:26:50 (6492): No heartbeat from core client for 30 sec - exiting 22:26:51 (6492): No heartbeat from core client for 30 sec - exiting 22:26:52 (6492): No heartbeat from core client for 30 sec - exiting 22:26:53 (6492): No heartbeat from core client for 30 sec - exiting 22:26:54 (6492): No heartbeat from core client for 30 sec - exiting 22:26:55 (6492): No heartbeat from core client for 30 sec - exiting 22:26:56 (6492): No heartbeat from core client for 30 sec - exiting 22:26:57 (6492): No heartbeat from core client for 30 sec - exiting 22:26:58 (6492): No heartbeat from core client for 30 sec - exiting 22:26:59 (6492): No heartbeat from core client for 30 sec - exiting 22:27:00 (6492): No heartbeat from core client for 30 sec - exiting 22:27:01 (6492): No heartbeat from core client for 30 sec - exiting 22:27:02 (6492): No heartbeat from core client for 30 sec - exiting 22:27:03 (6492): No heartbeat from core client for 30 sec - exiting 22:27:04 (6492): No heartbeat from core client for 30 sec - exiting 22:27:05 (6492): No heartbeat from core client for 30 sec - exiting 22:27:06 (6492): No heartbeat from core client for 30 sec - exiting 22:27:07 (6492): No heartbeat from core client for 30 sec - exiting 22:27:08 (6492): No heartbeat from core client for 30 sec - exiting 22:27:09 (6492): No heartbeat from core client for 30 sec - exiting 22:27:10 (6492): No heartbeat from core client for 30 sec - exiting 22:27:11 (6492): No heartbeat from core client for 30 sec - exiting 22:27:12 (6492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:13 (6492): No heartbeat from core client for 30 sec - exiting 22:27:14 (6492): No heartbeat from core client for 30 sec - exiting 22:27:56 (11504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:53:05 (12264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:28:41 (7900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:28:42 (7900): No heartbeat from core client for 30 sec - exiting 15:28:44 (7900): No heartbeat from core client for 30 sec - exiting 15:28:45 (7900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 00:18:23 (9164): No heartbeat from core client for 30 sec - exiting 00:18:41 (9164): No heartbeat from core client for 30 sec - exiting 00:18:42 (9164): No heartbeat from core client for 30 sec - exiting 00:18:43 (9164): No heartbeat from core client for 30 sec - exiting 00:18:44 (9164): No heartbeat from core client for 30 sec - exiting 00:18:45 (9164): No heartbeat from core client for 30 sec - exiting 00:18:46 (9164): No heartbeat from core client for 30 sec - exiting 00:18:47 (9164): No heartbeat from core client for 30 sec - exiting 00:18:48 (9164): No heartbeat from core client for 30 sec - exiting 00:18:49 (9164): No heartbeat from core client for 30 sec - exiting 00:18:50 (9164): No heartbeat from core client for 30 sec - exiting 00:18:51 (9164): No heartbeat from core client for 30 sec - exiting 00:18:52 (9164): No heartbeat from core client for 30 sec - exiting 00:18:53 (9164): No heartbeat from core client for 30 sec - exiting 00:18:54 (9164): No heartbeat from core client for 30 sec - exiting 00:18:55 (9164): No heartbeat from core client for 30 sec - exiting 00:18:56 (9164): No heartbeat from core client for 30 sec - exiting 00:18:57 (9164): No heartbeat from core client for 30 sec - exiting 00:18:58 (9164): No heartbeat from core client for 30 sec - exiting 00:18:59 (9164): No heartbeat from core client for 30 sec - exiting 00:19:00 (9164): No heartbeat from core client for 30 sec - exiting 00:19:01 (9164): No heartbeat from core client for 30 sec - exiting 00:19:02 (9164): No heartbeat from core client for 30 sec - exiting 00:19:03 (9164): No heartbeat from core client for 30 sec - exiting 00:19:04 (9164): No heartbeat from core client for 30 sec - exiting 00:19:05 (9164): No heartbeat from core client for 30 sec - exiting 00:19:06 (9164): No heartbeat from core client for 30 sec - exiting 00:19:07 (9164): No heartbeat from core client for 30 sec - exiting 00:19:09 (9164): No heartbeat from core client for 30 sec - exiting 00:19:10 (9164): No heartbeat from core client for 30 sec - exiting 00:19:11 (9164): No heartbeat from core client for 30 sec - exiting 00:19:12 (9164): No heartbeat from core client for 30 sec - exiting 00:19:13 (9164): No heartbeat from core client for 30 sec - exiting 00:19:14 (9164): No heartbeat from core client for 30 sec - exiting 00:19:15 (9164): No heartbeat from core client for 30 sec - exiting 00:19:16 (9164): No heartbeat from core client for 30 sec - exiting 00:19:17 (9164): No heartbeat from core client for 30 sec - exiting 00:19:18 (9164): No heartbeat from core client for 30 sec - exiting 00:19:19 (9164): No heartbeat from core client for 30 sec - exiting 00:19:20 (9164): No heartbeat from core client for 30 sec - exiting 00:19:21 (9164): No heartbeat from core client for 30 sec - exiting 00:19:22 (9164): No heartbeat from core client for 30 sec - exiting 00:19:23 (9164): No heartbeat from core client for 30 sec - exiting 00:19:24 (9164): No heartbeat from core client for 30 sec - exiting 00:19:25 (9164): No heartbeat from core client for 30 sec - exiting 00:19:26 (9164): No heartbeat from core client for 30 sec - exiting 00:19:27 (9164): No heartbeat from core client for 30 sec - exiting 00:19:28 (9164): No heartbeat from core client for 30 sec - exiting 00:19:30 (9164): No heartbeat from core client for 30 sec - exiting 00:19:31 (9164): No heartbeat from core client for 30 sec - exiting 00:19:32 (9164): No heartbeat from core client for 30 sec - exiting 00:19:33 (9164): No heartbeat from core client for 30 sec - exiting 00:19:34 (9164): No heartbeat from core client for 30 sec - exiting 00:19:35 (9164): No heartbeat from core client for 30 sec - exiting 00:19:36 (9164): No heartbeat from core client for 30 sec - exiting 00:19:37 (9164): No heartbeat from core client for 30 sec - exiting 00:19:38 (9164): No heartbeat from core client for 30 sec - exiting 00:19:39 (9164): No heartbeat from core client for 30 sec - exiting 00:19:40 (9164): No heartbeat from core client for 30 sec - exiting 00:19:41 (9164): No heartbeat from core client for 30 sec - exiting 00:19:42 (9164): No heartbeat from core client for 30 sec - exiting 00:19:43 (9164): No heartbeat from core client for 30 sec - exiting 00:19:44 (9164): No heartbeat from core client for 30 sec - exiting 00:19:45 (9164): No heartbeat from core client for 30 sec - exiting 00:19:46 (9164): No heartbeat from core client for 30 sec - exiting 00:19:47 (9164): No heartbeat from core client for 30 sec - exiting 00:19:48 (9164): No heartbeat from core client for 30 sec - exiting 00:19:49 (9164): No heartbeat from core client for 30 sec - exiting 00:19:50 (9164): No heartbeat from core client for 30 sec - exiting 00:19:51 (9164): No heartbeat from core client for 30 sec - exiting 00:19:52 (9164): No heartbeat from core client for 30 sec - exiting 00:19:53 (9164): No heartbeat from core client for 30 sec - exiting 00:19:54 (9164): No heartbeat from core client for 30 sec - exiting 00:19:55 (9164): No heartbeat from core client for 30 sec - exiting 00:19:56 (9164): No heartbeat from core client for 30 sec - exiting 00:19:57 (9164): No heartbeat from core client for 30 sec - exiting 00:19:58 (9164): No heartbeat from core client for 30 sec - exiting 00:19:59 (9164): No heartbeat from core client for 30 sec - exiting 00:20:00 (9164): No heartbeat from core client for 30 sec - exiting 00:20:01 (9164): No heartbeat from core client for 30 sec - exiting 00:20:02 (9164): No heartbeat from core client for 30 sec - exiting 00:20:03 (9164): No heartbeat from core client for 30 sec - exiting 00:20:04 (9164): No heartbeat from core client for 30 sec - exiting 00:20:05 (9164): No heartbeat from core client for 30 sec - exiting 00:20:06 (9164): No heartbeat from core client for 30 sec - exiting 00:20:07 (9164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:20:08 (9164): No heartbeat from core client for 30 sec - exiting 00:20:09 (9164): No heartbeat from core client for 30 sec - exiting 00:21:38 (11124): No heartbeat from core client for 30 sec - exiting 00:21:39 (11124): No heartbeat from core client for 30 sec - exiting 00:21:40 (11124): No heartbeat from core client for 30 sec - exiting 00:21:41 (11124): No heartbeat from core client for 30 sec - exiting 00:21:42 (11124): No heartbeat from core client for 30 sec - exiting 00:21:43 (11124): No heartbeat from core client for 30 sec - exiting 00:21:44 (11124): No heartbeat from core client for 30 sec - exiting 00:21:45 (11124): No heartbeat from core client for 30 sec - exiting 00:21:46 (11124): No heartbeat from core client for 30 sec - exiting 00:21:47 (11124): No heartbeat from core client for 30 sec - exiting 00:21:48 (11124): No heartbeat from core client for 30 sec - exiting 00:21:49 (11124): No heartbeat from core client for 30 sec - exiting 00:21:50 (11124): No heartbeat from core client for 30 sec - exiting 00:21:51 (11124): No heartbeat from core client for 30 sec - exiting 00:21:52 (11124): No heartbeat from core client for 30 sec - exiting 00:21:53 (11124): No heartbeat from core client for 30 sec - exiting 00:21:54 (11124): No heartbeat from core client for 30 sec - exiting 00:21:55 (11124): No heartbeat from core client for 30 sec - exiting 00:21:56 (11124): No heartbeat from core client for 30 sec - exiting 00:21:57 (11124): No heartbeat from core client for 30 sec - exiting 00:21:58 (11124): No heartbeat from core client for 30 sec - exiting 00:21:59 (11124): No heartbeat from core client for 30 sec - exiting 00:22:00 (11124): No heartbeat from core client for 30 sec - exiting 00:22:01 (11124): No heartbeat from core client for 30 sec - exiting 00:22:03 (11124): No heartbeat from core client for 30 sec - exiting 00:22:04 (11124): No heartbeat from core client for 30 sec - exiting 00:22:05 (11124): No heartbeat from core client for 30 sec - exiting 00:22:06 (11124): No heartbeat from core client for 30 sec - exiting 00:22:07 (11124): No heartbeat from core client for 30 sec - exiting 00:22:08 (11124): No heartbeat from core client for 30 sec - exiting 00:22:09 (11124): No heartbeat from core client for 30 sec - exiting 00:22:10 (11124): No heartbeat from core client for 30 sec - exiting 00:22:11 (11124): No heartbeat from core client for 30 sec - exiting 00:22:12 (11124): No heartbeat from core client for 30 sec - exiting 00:22:13 (11124): No heartbeat from core client for 30 sec - exiting 00:22:14 (11124): No heartbeat from core client for 30 sec - exiting 00:22:15 (11124): No heartbeat from core client for 30 sec - exiting 00:22:16 (11124): No heartbeat from core client for 30 sec - exiting 00:22:17 (11124): No heartbeat from core client for 30 sec - exiting 00:22:18 (11124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:22:19 (11124): No heartbeat from core client for 30 sec - exiting 01:41:35 (8460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:41:36 (8460): No heartbeat from core client for 30 sec - exiting 01:44:40 (11368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:48:25 (6312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5072, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jul 2011 15:40:32	1108969	12991861	hadcm3n_s6fl_1940_40_007300538_0	233,280	570,726	2.4465
05 Jul 2011 16:12:23	1108969	12991861	hadcm3n_s6fl_1940_40_007300538_0	207,360	505,909	2.4398
03 Jul 2011 13:57:13	1108969	12991861	hadcm3n_s6fl_1940_40_007300538_0	181,440	445,874	2.4574
02 Jul 2011 04:48:20	1108969	12991861	hadcm3n_s6fl_1940_40_007300538_0	155,520	383,785	2.4678
30 Jun 2011 07:03:48	1108969	12991861	hadcm3n_s6fl_1940_40_007300538_0	129,600	322,022	2.4847
28 Jun 2011 15:12:57	1108969	12991861	hadcm3n_s6fl_1940_40_007300538_0	103,680	258,277	2.4911
26 Jun 2011 05:24:28	1108969	12991861	hadcm3n_s6fl_1940_40_007300538_0	77,760	193,148	2.4839
24 Jun 2011 01:27:21	1108969	12991861	hadcm3n_s6fl_1940_40_007300538_0	51,840	124,123	2.3943
22 Jun 2011 12:08:16	1108969	12991861	hadcm3n_s6fl_1940_40_007300538_0	25,920	60,276	2.3255