Task 16062405

Name	hadcm3n_n3an_1880_40_008398733_2
Workunit	8549589
Created	9 Oct 2013, 3:24:19 UTC
Sent	9 Oct 2013, 3:42:58 UTC
Report deadline	8 Jan 2014, 11:10:09 UTC
Received	13 Nov 2013, 13:09:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1237173
Run time	3 days 5 hours 46 min 50 sec
CPU time	2 days 13 hours 39 min 12 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.30 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 23:06:42 (116832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:43:14 (102104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:43:18 (102104): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 00:15:26 (114020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:15:27 (114020): No heartbeat from core client for 30 sec - exiting 00:15:28 (114020): No heartbeat from core client for 30 sec - exiting 00:15:29 (114020): No heartbeat from core client for 30 sec - exiting 00:15:30 (114020): No heartbeat from core client for 30 sec - exiting 00:15:31 (114020): No heartbeat from core client for 30 sec - exiting 00:15:32 (114020): No heartbeat from core client for 30 sec - exiting 00:15:33 (114020): No heartbeat from core client for 30 sec - exiting 00:15:34 (114020): No heartbeat from core client for 30 sec - exiting 00:15:35 (114020): No heartbeat from core client for 30 sec - exiting 00:15:36 (114020): No heartbeat from core client for 30 sec - exiting 00:22:13 (108048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:22:14 (108048): No heartbeat from core client for 30 sec - exiting 00:22:15 (108048): No heartbeat from core client for 30 sec - exiting 00:22:16 (108048): No heartbeat from core client for 30 sec - exiting 00:22:17 (108048): No heartbeat from core client for 30 sec - exiting 00:22:18 (108048): No heartbeat from core client for 30 sec - exiting 00:22:19 (108048): No heartbeat from core client for 30 sec - exiting 00:22:20 (108048): No heartbeat from core client for 30 sec - exiting 00:22:21 (108048): No heartbeat from core client for 30 sec - exiting 00:22:22 (108048): No heartbeat from core client for 30 sec - exiting 00:22:23 (108048): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 00:49:53 (116536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:49:54 (116536): No heartbeat from core client for 30 sec - exiting 00:49:55 (116536): No heartbeat from core client for 30 sec - exiting 00:49:56 (116536): No heartbeat from core client for 30 sec - exiting 00:49:57 (116536): No heartbeat from core client for 30 sec - exiting 00:49:58 (116536): No heartbeat from core client for 30 sec - exiting 00:49:59 (116536): No heartbeat from core client for 30 sec - exiting 00:50:00 (116536): No heartbeat from core client for 30 sec - exiting 00:50:01 (116536): No heartbeat from core client for 30 sec - exiting 00:50:02 (116536): No heartbeat from core client for 30 sec - exiting 00:50:03 (116536): No heartbeat from core client for 30 sec - exiting 00:53:55 (113052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:54:14 (113052): No heartbeat from core client for 30 sec - exiting 00:54:15 (113052): No heartbeat from core client for 30 sec - exiting 00:54:16 (113052): No heartbeat from core client for 30 sec - exiting 00:54:17 (113052): No heartbeat from core client for 30 sec - exiting 00:54:18 (113052): No heartbeat from core client for 30 sec - exiting 00:54:19 (113052): No heartbeat from core client for 30 sec - exiting 00:54:20 (113052): No heartbeat from core client for 30 sec - exiting 00:54:21 (113052): No heartbeat from core client for 30 sec - exiting 00:54:22 (113052): No heartbeat from core client for 30 sec - exiting 00:54:23 (113052): No heartbeat from core client for 30 sec - exiting 01:31:12 (107608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:10:50 (106596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:02 (106596): No heartbeat from core client for 30 sec - exiting 04:11:03 (106596): No heartbeat from core client for 30 sec - exiting 04:11:04 (106596): No heartbeat from core client for 30 sec - exiting 04:11:05 (106596): No heartbeat from core client for 30 sec - exiting 04:11:06 (106596): No heartbeat from core client for 30 sec - exiting 04:11:07 (106596): No heartbeat from core client for 30 sec - exiting 04:11:08 (106596): No heartbeat from core client for 30 sec - exiting 04:11:09 (106596): No heartbeat from core client for 30 sec - exiting 04:11:10 (106596): No heartbeat from core client for 30 sec - exiting 06:03:05 (102032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:03:07 (102032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 09:25:09 (114432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:25:25 (114432): No heartbeat from core client for 30 sec - exiting 09:30:04 (118100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:41:09 (98536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:02:20 (103620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:50:04 (112280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:50:07 (112280): No heartbeat from core client for 30 sec - exiting 14:50:08 (112280): No heartbeat from core client for 30 sec - exiting 14:50:09 (112280): No heartbeat from core client for 30 sec - exiting 14:50:10 (112280): No heartbeat from core client for 30 sec - exiting 14:50:11 (112280): No heartbeat from core client for 30 sec - exiting 14:50:12 (112280): No heartbeat from core client for 30 sec - exiting 14:50:13 (112280): No heartbeat from core client for 30 sec - exiting 14:50:14 (112280): No heartbeat from core client for 30 sec - exiting 14:50:15 (112280): No heartbeat from core client for 30 sec - exiting 14:50:16 (112280): No heartbeat from core client for 30 sec - exiting 14:50:17 (112280): No heartbeat from core client for 30 sec - exiting 16:08:13 (112648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:08:26 (112648): No heartbeat from core client for 30 sec - exiting 16:08:27 (112648): No heartbeat from core client for 30 sec - exiting 16:08:28 (112648): No heartbeat from core client for 30 sec - exiting 16:08:29 (112648): No heartbeat from core client for 30 sec - exiting 16:08:30 (112648): No heartbeat from core client for 30 sec - exiting 16:08:31 (112648): No heartbeat from core client for 30 sec - exiting 16:46:02 (101064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:53:35 (104268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:56:40 (105372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:03:45 (115776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:03:46 (115776): No heartbeat from core client for 30 sec - exiting 17:03:47 (115776): No heartbeat from core client for 30 sec - exiting 17:03:48 (115776): No heartbeat from core client for 30 sec - exiting 17:03:49 (115776): No heartbeat from core client for 30 sec - exiting 17:03:50 (115776): No heartbeat from core client for 30 sec - exiting 17:03:51 (115776): No heartbeat from core client for 30 sec - exiting 17:03:53 (115776): No heartbeat from core client for 30 sec - exiting 17:03:56 (115776): No heartbeat from core client for 30 sec - exiting 17:03:57 (115776): No heartbeat from core client for 30 sec - exiting 17:03:58 (115776): No heartbeat from core client for 30 sec - exiting 17:03:59 (115776): No heartbeat from core client for 30 sec - exiting 17:04:00 (115776): No heartbeat from core client for 30 sec - exiting 17:04:01 (115776): No heartbeat from core client for 30 sec - exiting 17:04:02 (115776): No heartbeat from core client for 30 sec - exiting 17:04:03 (115776): No heartbeat from core client for 30 sec - exiting 17:04:04 (115776): No heartbeat from core client for 30 sec - exiting 17:04:05 (115776): No heartbeat from core client for 30 sec - exiting 17:04:06 (115776): No heartbeat from core client for 30 sec - exiting 17:04:07 (115776): No heartbeat from core client for 30 sec - exiting 17:04:08 (115776): No heartbeat from core client for 30 sec - exiting 17:04:09 (115776): No heartbeat from core client for 30 sec - exiting 17:04:10 (115776): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 19:07:12 (102432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:07:29 (102432): No heartbeat from core client for 30 sec - exiting 19:07:32 (102432): No heartbeat from core client for 30 sec - exiting 19:07:33 (102432): No heartbeat from core client for 30 sec - exiting forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=103648, iMonCtr=1 Model crash detected, will try to restart... 08:13:00 (103648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:01 (103648): No heartbeat from core client for 30 sec - exiting 08:13:02 (103648): No heartbeat from core client for 30 sec - exiting 08:13:03 (103648): No heartbeat from core client for 30 sec - exiting 08:13:04 (103648): No heartbeat from core client for 30 sec - exiting 08:13:05 (103648): No heartbeat from core client for 30 sec - exiting 08:13:06 (103648): No heartbeat from core client for 30 sec - exiting 08:13:07 (103648): No heartbeat from core client for 30 sec - exiting 08:13:08 (103648): No heartbeat from core client for 30 sec - exiting 02:28:17 (8360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:28:21 (8360): No heartbeat from core client for 30 sec - exiting 02:28:22 (8360): No heartbeat from core client for 30 sec - exiting 02:28:24 (8360): No heartbeat from core client for 30 sec - exiting 05:30:58 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 06:34:16 (7068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:34:17 (7068): No heartbeat from core client for 30 sec - exiting 06:34:18 (7068): No heartbeat from core client for 30 sec - exiting 06:34:19 (7068): No heartbeat from core client for 30 sec - exiting 06:34:20 (7068): No heartbeat from core client for 30 sec - exiting 06:34:21 (7068): No heartbeat from core client for 30 sec - exiting 02:04:45 (9308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:04:52 (9308): No heartbeat from core client for 30 sec - exiting 02:04:53 (9308): No heartbeat from core client for 30 sec - exiting 02:04:54 (9308): No heartbeat from core client for 30 sec - exiting 02:04:55 (9308): No heartbeat from core client for 30 sec - exiting 02:04:56 (9308): No heartbeat from core client for 30 sec - exiting 02:04:57 (9308): No heartbeat from core client for 30 sec - exiting 02:04:58 (9308): No heartbeat from core client for 30 sec - exiting 02:04:59 (9308): No heartbeat from core client for 30 sec - exiting 02:05:00 (9308): No heartbeat from core client for 30 sec - exiting 02:07:03 (15472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:07:04 (15472): No heartbeat from core client for 30 sec - exiting 02:07:05 (15472): No heartbeat from core client for 30 sec - exiting 02:07:06 (15472): No heartbeat from core client for 30 sec - exiting 02:07:07 (15472): No heartbeat from core client for 30 sec - exiting 02:07:08 (15472): No heartbeat from core client for 30 sec - exiting 02:07:09 (15472): No heartbeat from core client for 30 sec - exiting 02:07:10 (15472): No heartbeat from core client for 30 sec - exiting 02:07:11 (15472): No heartbeat from core client for 30 sec - exiting 02:07:12 (15472): No heartbeat from core client for 30 sec - exiting 02:07:13 (15472): No heartbeat from core client for 30 sec - exiting 03:29:29 (16264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:29:30 (16264): No heartbeat from core client for 30 sec - exiting 03:29:31 (16264): No heartbeat from core client for 30 sec - exiting 03:29:32 (16264): No heartbeat from core client for 30 sec - exiting 03:29:33 (16264): No heartbeat from core client for 30 sec - exiting 03:29:34 (16264): No heartbeat from core client for 30 sec - exiting 03:29:36 (16264): No heartbeat from core client for 30 sec - exiting 03:29:37 (16264): No heartbeat from core client for 30 sec - exiting 03:29:38 (16264): No heartbeat from core client for 30 sec - exiting 03:29:39 (16264): No heartbeat from core client for 30 sec - exiting 03:29:40 (16264): No heartbeat from core client for 30 sec - exiting 03:37:50 (16932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:40:33 (12712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:40:51 (12712): No heartbeat from core client for 30 sec - exiting 03:40:52 (12712): No heartbeat from core client for 30 sec - exiting 03:40:53 (12712): No heartbeat from core client for 30 sec - exiting 03:40:54 (12712): No heartbeat from core client for 30 sec - exiting 03:40:55 (12712): No heartbeat from core client for 30 sec - exiting 03:40:56 (12712): No heartbeat from core client for 30 sec - exiting 03:40:57 (12712): No heartbeat from core client for 30 sec - exiting 03:40:58 (12712): No heartbeat from core client for 30 sec - exiting 03:40:59 (12712): No heartbeat from core client for 30 sec - exiting 03:41:00 (12712): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 03:44:01 (17384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:44:02 (17384): No heartbeat from core client for 30 sec - exiting 03:44:03 (17384): No heartbeat from core client for 30 sec - exiting 03:44:04 (17384): No heartbeat from core client for 30 sec - exiting 03:44:05 (17384): No heartbeat from core client for 30 sec - exiting 03:44:06 (17384): No heartbeat from core client for 30 sec - exiting 03:44:07 (17384): No heartbeat from core client for 30 sec - exiting 03:44:08 (17384): No heartbeat from core client for 30 sec - exiting 03:44:09 (17384): No heartbeat from core client for 30 sec - exiting 03:44:10 (17384): No heartbeat from core client for 30 sec - exiting 03:44:11 (17384): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 03:48:52 (14056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:48:53 (14056): No heartbeat from core client for 30 sec - exiting 03:48:54 (14056): No heartbeat from core client for 30 sec - exiting 03:48:55 (14056): No heartbeat from core client for 30 sec - exiting 03:48:56 (14056): No heartbeat from core client for 30 sec - exiting 03:48:57 (14056): No heartbeat from core client for 30 sec - exiting 03:48:58 (14056): No heartbeat from core client for 30 sec - exiting 04:09:31 (12872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:10 (8176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:11 (8176): No heartbeat from core client for 30 sec - exiting 04:11:12 (8176): No heartbeat from core client for 30 sec - exiting 04:11:13 (8176): No heartbeat from core client for 30 sec - exiting 04:11:14 (8176): No heartbeat from core client for 30 sec - exiting 04:11:15 (8176): No heartbeat from core client for 30 sec - exiting 04:11:16 (8176): No heartbeat from core client for 30 sec - exiting 04:11:17 (8176): No heartbeat from core client for 30 sec - exiting 04:11:18 (8176): No heartbeat from core client for 30 sec - exiting 04:11:19 (8176): No heartbeat from core client for 30 sec - exiting 04:11:20 (8176): No heartbeat from core client for 30 sec - exiting 04:46:17 (16448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:46:19 (16448): No heartbeat from core client for 30 sec - exiting 04:46:57 (13660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:47:12 (13660): No heartbeat from core client for 30 sec - exiting 04:47:13 (13660): No heartbeat from core client for 30 sec - exiting 04:47:14 (13660): No heartbeat from core client for 30 sec - exiting 04:47:15 (13660): No heartbeat from core client for 30 sec - exiting 04:47:16 (13660): No heartbeat from core client for 30 sec - exiting 04:47:17 (13660): No heartbeat from core client for 30 sec - exiting 04:47:18 (13660): No heartbeat from core client for 30 sec - exiting 04:47:19 (13660): No heartbeat from core client for 30 sec - exiting 04:47:20 (13660): No heartbeat from core client for 30 sec - exiting 04:47:21 (13660): No heartbeat from core client for 30 sec - exiting 05:15:07 (6816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:22:52 (16944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6992, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Nov 2013 06:15:44	1237173	16062405	hadcm3n_n3an_1880_40_008398733_2	259,200	206,967	0.7985
13 Nov 2013 00:54:32	1237173	16062405	hadcm3n_n3an_1880_40_008398733_2	233,280	188,642	0.8087
12 Nov 2013 19:48:00	1237173	16062405	hadcm3n_n3an_1880_40_008398733_2	207,360	170,362	0.8216
12 Nov 2013 14:36:12	1237173	16062405	hadcm3n_n3an_1880_40_008398733_2	181,440	152,118	0.8384
12 Nov 2013 09:32:04	1237173	16062405	hadcm3n_n3an_1880_40_008398733_2	155,520	133,876	0.8608
12 Nov 2013 07:06:24	1237173	16062405	hadcm3n_n3an_1880_40_008398733_2	129,600	114,770	0.8856
12 Nov 2013 07:06:24	1237173	16062405	hadcm3n_n3an_1880_40_008398733_2	103,680	95,197	0.9182
12 Nov 2013 07:06:24	1237173	16062405	hadcm3n_n3an_1880_40_008398733_2	77,760	75,616	0.9724
12 Nov 2013 07:06:24	1237173	16062405	hadcm3n_n3an_1880_40_008398733_2	51,840	55,902	1.0784
12 Nov 2013 07:06:24	1237173	16062405	hadcm3n_n3an_1880_40_008398733_2	25,920	28,581	1.1027