Task 15457344

Name	hadcm3n_zlb8_1880_40_008252531_1
Workunit	8407655
Created	23 Nov 2012, 14:14:45 UTC
Sent	23 Nov 2012, 14:15:11 UTC
Report deadline	22 Feb 2013, 21:42:22 UTC
Received	4 Dec 2012, 11:34:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1246636
Run time	10 days 0 hours 5 min 3 sec
CPU time	8 days 18 hours 4 min 19 sec
Validate state	Invalid
Credit	8,087.04
Device peak FLOPS	3.43 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 14:47:11 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:52:11 (8436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:08:59 (924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:00 (924): No heartbeat from core client for 30 sec - exiting 07:09:01 (924): No heartbeat from core client for 30 sec - exiting 07:09:02 (924): No heartbeat from core client for 30 sec - exiting 07:09:03 (924): No heartbeat from core client for 30 sec - exiting 07:09:04 (924): No heartbeat from core client for 30 sec - exiting 07:09:05 (924): No heartbeat from core client for 30 sec - exiting 07:09:06 (924): No heartbeat from core client for 30 sec - exiting 07:09:07 (924): No heartbeat from core client for 30 sec - exiting 07:09:08 (924): No heartbeat from core client for 30 sec - exiting 07:09:09 (924): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 07:19:01 (5972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:19:02 (5972): No heartbeat from core client for 30 sec - exiting 07:19:03 (5972): No heartbeat from core client for 30 sec - exiting 07:19:04 (5972): No heartbeat from core client for 30 sec - exiting 07:19:05 (5972): No heartbeat from core client for 30 sec - exiting 07:19:06 (5972): No heartbeat from core client for 30 sec - exiting 07:19:07 (5972): No heartbeat from core client for 30 sec - exiting 07:19:08 (5972): No heartbeat from core client for 30 sec - exiting 07:19:09 (5972): No heartbeat from core client for 30 sec - exiting 07:19:10 (5972): No heartbeat from core client for 30 sec - exiting 07:19:11 (5972): No heartbeat from core client for 30 sec - exiting 09:47:41 (3216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:47:42 (3216): No heartbeat from core client for 30 sec - exiting 09:47:43 (3216): No heartbeat from core client for 30 sec - exiting 09:47:44 (3216): No heartbeat from core client for 30 sec - exiting 09:47:45 (3216): No heartbeat from core client for 30 sec - exiting 09:47:46 (3216): No heartbeat from core client for 30 sec - exiting 09:47:47 (3216): No heartbeat from core client for 30 sec - exiting 09:47:48 (3216): No heartbeat from core client for 30 sec - exiting 09:47:49 (3216): No heartbeat from core client for 30 sec - exiting 09:47:50 (3216): No heartbeat from core client for 30 sec - exiting 09:47:51 (3216): No heartbeat from core client for 30 sec - exiting 13:20:18 (6536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:21:27 (7864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:21:28 (7864): No heartbeat from core client for 30 sec - exiting 21:21:29 (7864): No heartbeat from core client for 30 sec - exiting 21:21:30 (7864): No heartbeat from core client for 30 sec - exiting 21:21:31 (7864): No heartbeat from core client for 30 sec - exiting 21:21:32 (7864): No heartbeat from core client for 30 sec - exiting 21:21:33 (7864): No heartbeat from core client for 30 sec - exiting 21:21:34 (7864): No heartbeat from core client for 30 sec - exiting 21:21:35 (7864): No heartbeat from core client for 30 sec - exiting 21:21:36 (7864): No heartbeat from core client for 30 sec - exiting 21:21:37 (7864): No heartbeat from core client for 30 sec - exiting 22:01:57 (6676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:01:58 (6676): No heartbeat from core client for 30 sec - exiting 22:01:59 (6676): No heartbeat from core client for 30 sec - exiting 22:02:00 (6676): No heartbeat from core client for 30 sec - exiting 22:02:01 (6676): No heartbeat from core client for 30 sec - exiting 22:06:58 (3884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:17:05 (8292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 00:27:05 (8240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:27:06 (8240): No heartbeat from core client for 30 sec - exiting 00:27:07 (8240): No heartbeat from core client for 30 sec - exiting 00:27:08 (8240): No heartbeat from core client for 30 sec - exiting 00:27:09 (8240): No heartbeat from core client for 30 sec - exiting 00:27:10 (8240): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 02:52:18 (6932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:52:19 (6932): No heartbeat from core client for 30 sec - exiting 02:52:20 (6932): No heartbeat from core client for 30 sec - exiting 02:52:21 (6932): No heartbeat from core client for 30 sec - exiting 02:52:22 (6932): No heartbeat from core client for 30 sec - exiting 02:52:23 (6932): No heartbeat from core client for 30 sec - exiting 02:52:24 (6932): No heartbeat from core client for 30 sec - exiting 02:52:25 (6932): No heartbeat from core client for 30 sec - exiting 02:52:26 (6932): No heartbeat from core client for 30 sec - exiting 02:52:27 (6932): No heartbeat from core client for 30 sec - exiting 02:52:28 (6932): No heartbeat from core client for 30 sec - exiting 13:41:49 (3804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:41:50 (3804): No heartbeat from core client for 30 sec - exiting 13:41:51 (3804): No heartbeat from core client for 30 sec - exiting 13:41:52 (3804): No heartbeat from core client for 30 sec - exiting 13:41:53 (3804): No heartbeat from core client for 30 sec - exiting 13:41:54 (3804): No heartbeat from core client for 30 sec - exiting 13:41:55 (3804): No heartbeat from core client for 30 sec - exiting 13:41:56 (3804): No heartbeat from core client for 30 sec - exiting 13:41:57 (3804): No heartbeat from core client for 30 sec - exiting 13:41:58 (3804): No heartbeat from core client for 30 sec - exiting 13:41:59 (3804): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 16:53:04 (6452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:53:05 (6452): No heartbeat from core client for 30 sec - exiting 16:53:06 (6452): No heartbeat from core client for 30 sec - exiting 16:53:07 (6452): No heartbeat from core client for 30 sec - exiting 16:53:08 (6452): No heartbeat from core client for 30 sec - exiting 16:53:09 (6452): No heartbeat from core client for 30 sec - exiting 16:53:10 (6452): No heartbeat from core client for 30 sec - exiting 16:53:11 (6452): No heartbeat from core client for 30 sec - exiting 16:53:12 (6452): No heartbeat from core client for 30 sec - exiting 16:53:13 (6452): No heartbeat from core client for 30 sec - exiting 16:53:14 (6452): No heartbeat from core client for 30 sec - exiting 17:06:10 (8504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:00:01 (5612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:11:04 (4896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:33:54 (5748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:33:55 (5748): No heartbeat from core client for 30 sec - exiting 08:33:56 (5748): No heartbeat from core client for 30 sec - exiting 08:33:57 (5748): No heartbeat from core client for 30 sec - exiting 08:33:58 (5748): No heartbeat from core client for 30 sec - exiting 08:33:59 (5748): No heartbeat from core client for 30 sec - exiting 08:34:00 (5748): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 22:32:06 (4148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:32:07 (4148): No heartbeat from core client for 30 sec - exiting 22:32:08 (4148): No heartbeat from core client for 30 sec - exiting 22:32:09 (4148): No heartbeat from core client for 30 sec - exiting 22:32:10 (4148): No heartbeat from core client for 30 sec - exiting 22:32:11 (4148): No heartbeat from core client for 30 sec - exiting 22:32:12 (4148): No heartbeat from core client for 30 sec - exiting 22:32:13 (4148): No heartbeat from core client for 30 sec - exiting 22:32:14 (4148): No heartbeat from core client for 30 sec - exiting 22:32:15 (4148): No heartbeat from core client for 30 sec - exiting 22:32:16 (4148): No heartbeat from core client for 30 sec - exiting 02:02:24 (8568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 04:50:17 (9476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:50:18 (9476): No heartbeat from core client for 30 sec - exiting 04:50:19 (9476): No heartbeat from core client for 30 sec - exiting 04:50:20 (9476): No heartbeat from core client for 30 sec - exiting 04:50:21 (9476): No heartbeat from core client for 30 sec - exiting 04:50:22 (9476): No heartbeat from core client for 30 sec - exiting 04:50:23 (9476): No heartbeat from core client for 30 sec - exiting 04:50:24 (9476): No heartbeat from core client for 30 sec - exiting 04:50:25 (9476): No heartbeat from core client for 30 sec - exiting 04:50:26 (9476): No heartbeat from core client for 30 sec - exiting 04:50:27 (9476): No heartbeat from core client for 30 sec - exiting 07:22:16 (4100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:24:47 (6640): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:23:06 (7528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:23:07 (7528): No heartbeat from core client for 30 sec - exiting 19:23:08 (7528): No heartbeat from core client for 30 sec - exiting 19:23:09 (7528): No heartbeat from core client for 30 sec - exiting 19:23:10 (7528): No heartbeat from core client for 30 sec - exiting 19:23:11 (7528): No heartbeat from core client for 30 sec - exiting 19:23:12 (7528): No heartbeat from core client for 30 sec - exiting 19:23:13 (7528): No heartbeat from core client for 30 sec - exiting 19:23:14 (7528): No heartbeat from core client for 30 sec - exiting 19:23:15 (7528): No heartbeat from core client for 30 sec - exiting 19:23:16 (7528): No heartbeat from core client for 30 sec - exiting 19:28:06 (7344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:48:14 (6036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 00:22:29 (8244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:22:30 (8244): No heartbeat from core client for 30 sec - exiting 00:22:31 (8244): No heartbeat from core client for 30 sec - exiting 00:22:32 (8244): No heartbeat from core client for 30 sec - exiting 00:22:33 (8244): No heartbeat from core client for 30 sec - exiting 00:22:34 (8244): No heartbeat from core client for 30 sec - exiting 00:22:35 (8244): No heartbeat from core client for 30 sec - exiting 00:22:36 (8244): No heartbeat from core client for 30 sec - exiting 00:22:37 (8244): No heartbeat from core client for 30 sec - exiting 00:22:38 (8244): No heartbeat from core client for 30 sec - exiting 00:22:39 (8244): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 04:43:17 (2936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:01 (9248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:02 (9248): No heartbeat from core client for 30 sec - exiting 07:09:03 (9248): No heartbeat from core client for 30 sec - exiting 07:09:04 (9248): No heartbeat from core client for 30 sec - exiting 07:09:05 (9248): No heartbeat from core client for 30 sec - exiting 07:09:06 (9248): No heartbeat from core client for 30 sec - exiting 07:09:07 (9248): No heartbeat from core client for 30 sec - exiting 07:09:08 (9248): No heartbeat from core client for 30 sec - exiting 07:09:09 (9248): No heartbeat from core client for 30 sec - exiting 07:09:10 (9248): No heartbeat from core client for 30 sec - exiting 07:09:11 (9248): No heartbeat from core client for 30 sec - exiting 07:21:11 (7464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6660, iMonCtr=1 Model crash detected, will try to restart... 08:39:01 (5520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:03:47 (2384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 00:32:44 (6364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:16:34 (5380):No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3556, selfPID=3556, iMonCtr=1 CPDN Monitor - Quit request from BOINC... 15:09:45 (5156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:09:51 (5156): No heartbeat from core client for 30 sec - exiting 15:09:52 (5156): No heartbeat from core client for 30 sec - exiting 15:09:53 (5156): No heartbeat from core client for 30 sec - exiting 15:10:40 (6972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Dec 2012 19:37:03	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	673,920	745,358	1.1060
03 Dec 2012 10:04:37	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	648,000	717,389	1.1071
03 Dec 2012 00:22:44	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	622,080	688,973	1.1075
02 Dec 2012 15:39:54	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	596,160	660,380	1.1077
02 Dec 2012 06:13:09	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	570,240	631,465	1.1074
01 Dec 2012 20:54:39	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	544,320	602,865	1.1076
01 Dec 2012 13:22:19	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	518,400	574,438	1.1081
01 Dec 2012 04:10:17	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	492,480	545,765	1.1082
30 Nov 2012 19:21:20	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	466,560	517,218	1.1086
30 Nov 2012 10:47:20	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	440,640	488,044	1.1076
30 Nov 2012 02:00:23	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	414,720	458,358	1.1052
29 Nov 2012 17:13:56	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	388,800	428,882	1.1031
29 Nov 2012 08:17:22	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	362,880	398,986	1.0995
28 Nov 2012 23:20:29	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	336,960	370,724	1.1002
28 Nov 2012 14:03:02	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	311,040	342,257	1.1004
28 Nov 2012 02:40:21	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	285,120	313,698	1.1002
27 Nov 2012 17:55:54	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	259,200	285,333	1.1008
27 Nov 2012 07:13:20	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	233,280	256,359	1.0989
26 Nov 2012 20:09:23	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	207,360	227,889	1.0990
26 Nov 2012 10:30:30	1246636	15457344	hadcm3n_zlb8_1880_40_008252531_1	181,440	199,290	1.0984