Task 15880702

Name	hadcm3n_n1hp_1920_40_008389774_2
Workunit	8540633
Created	4 Jul 2013, 12:05:05 UTC
Sent	4 Jul 2013, 12:59:03 UTC
Report deadline	3 Oct 2013, 20:26:14 UTC
Received	14 Aug 2013, 20:48:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1129647
Run time	13 days 0 hours 54 min 16 sec
CPU time	12 days 7 hours 21 min 8 sec
Validate state	Invalid
Credit	6,842.88
Device peak FLOPS	2.91 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5412, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:56:28 (5464): No heartbeat from core client for 30 sec - exiting 07:56:29 (5464): No heartbeat from core client for 30 sec - exiting 07:56:30 (5464): No heartbeat from core client for 30 sec - exiting 07:56:31 (5464): No heartbeat from core client for 30 sec - exiting 07:56:32 (5464): No heartbeat from core client for 30 sec - exiting 07:56:33 (5464): No heartbeat from core client for 30 sec - exiting 07:56:34 (5464): No heartbeat from core client for 30 sec - exiting 07:56:35 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:56:36 (5464): No heartbeat from core client for 30 sec - exiting 05:53:28 (5632): No heartbeat from core client for 30 sec - exiting 05:53:29 (5632): No heartbeat from core client for 30 sec - exiting 05:53:30 (5632): No heartbeat from core client for 30 sec - exiting 05:53:31 (5632): No heartbeat from core client for 30 sec - exiting 05:53:32 (5632): No heartbeat from core client for 30 sec - exiting 05:53:33 (5632): No heartbeat from core client for 30 sec - exiting 05:53:34 (5632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:08:19 (5552): No heartbeat from core client for 30 sec - exiting 17:08:21 (5552): No heartbeat from core client for 30 sec - exiting 17:08:22 (5552): No heartbeat from core client for 30 sec - exiting 17:08:23 (5552): No heartbeat from core client for 30 sec - exiting 17:08:24 (5552): No heartbeat from core client for 30 sec - exiting 17:08:25 (5552): No heartbeat from core client for 30 sec - exiting 17:08:26 (5552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1816, iMonCtr=1 Model crash detected, will try to restart... 05:35:55 (5604): No heartbeat from core client for 30 sec - exiting 05:35:57 (5604): No heartbeat from core client for 30 sec - exiting 05:35:58 (5604): No heartbeat from core client for 30 sec - exiting 05:35:59 (5604): No heartbeat from core client for 30 sec - exiting 05:36:00 (5604): No heartbeat from core client for 30 sec - exiting 05:36:01 (5604): No heartbeat from core client for 30 sec - exiting 05:36:02 (5604): No heartbeat from core client for 30 sec - exiting 05:36:03 (5604): No heartbeat from core client for 30 sec - exiting 05:36:04 (5604): No heartbeat from core client for 30 sec - exiting 05:36:05 (5604): No heartbeat from core client for 30 sec - exiting 05:36:06 (5604): No heartbeat from core client for 30 sec - exiting 05:36:07 (5604): No heartbeat from core client for 30 sec - exiting 05:36:08 (5604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6104, iMonCtr=1 Model crash detected, will try to restart... 16:30:41 (6004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:53:40 (5984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:42:29 (5644): No heartbeat from core client for 30 sec - exiting 05:42:31 (5644): No heartbeat from core client for 30 sec - exiting 05:42:32 (5644): No heartbeat from core client for 30 sec - exiting 05:42:33 (5644): No heartbeat from core client for 30 sec - exiting 05:42:34 (5644): No heartbeat from core client for 30 sec - exiting 05:42:35 (5644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:43:38 (5332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:37:42 (4184): No heartbeat from core client for 30 sec - exiting 05:37:43 (4184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:47:16 (5212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:13:14 (9608): No heartbeat from core client for 30 sec - exiting 16:13:15 (9608): No heartbeat from core client for 30 sec - exiting 16:13:16 (9608): No heartbeat from core client for 30 sec - exiting 16:13:17 (9608): No heartbeat from core client for 30 sec - exiting 16:13:18 (9608): No heartbeat from core client for 30 sec - exiting 16:13:19 (9608): No heartbeat from core client for 30 sec - exiting 16:13:20 (9608): No heartbeat from core client for 30 sec - exiting 16:13:21 (9608): No heartbeat from core client for 30 sec - exiting 16:13:22 (9608): No heartbeat from core client for 30 sec - exiting 16:13:23 (9608): No heartbeat from core client for 30 sec - exiting 16:13:24 (9608): No heartbeat from core client for 30 sec - exiting 16:13:25 (9608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:13:26 (9608): No heartbeat from core client for 30 sec - exiting 16:13:59 (1644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:44:37 (4588): No heartbeat from core client for 30 sec - exiting 16:44:38 (4588): No heartbeat from core client for 30 sec - exiting 16:44:39 (4588): No heartbeat from core client for 30 sec - exiting 16:44:40 (4588): No heartbeat from core client for 30 sec - exiting 16:44:41 (4588): No heartbeat from core client for 30 sec - exiting 16:44:42 (4588): No heartbeat from core client for 30 sec - exiting 16:44:43 (4588): No heartbeat from core client for 30 sec - exiting 16:44:44 (4588): No heartbeat from core client for 30 sec - exiting 16:44:45 (4588): No heartbeat from core client for 30 sec - exiting 16:44:46 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:43:58 (4280): No heartbeat from core client for 30 sec - exiting 16:43:59 (4280): No heartbeat from core client for 30 sec - exiting 16:44:00 (4280): No heartbeat from core client for 30 sec - exiting 16:44:01 (4280): No heartbeat from core client for 30 sec - exiting 16:44:02 (4280): No heartbeat from core client for 30 sec - exiting 16:44:03 (4280): No heartbeat from core client for 30 sec - exiting 16:44:04 (4280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:37:20 (5292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:37:21 (5292): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2712, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2712, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2712, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2712, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2712, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2712, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Aug 2013 20:50:12	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	570,240	1,024,673	1.7969
14 Aug 2013 20:50:12	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	544,320	979,013	1.7986
14 Aug 2013 20:50:12	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	518,400	932,983	1.7997
30 Jul 2013 10:21:58	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	492,480	885,869	1.7988
25 Jul 2013 11:06:07	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	466,560	839,521	1.7994
24 Jul 2013 09:36:25	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	440,640	792,024	1.7974
23 Jul 2013 22:06:16	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	414,720	745,647	1.7980
23 Jul 2013 21:27:49	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	388,800	699,396	1.7989
23 Jul 2013 20:42:25	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	362,880	653,567	1.8011
23 Jul 2013 20:01:31	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	336,960	606,842	1.8009
23 Jul 2013 19:08:20	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	311,040	560,770	1.8029
23 Jul 2013 14:26:59	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	285,120	513,801	1.8021
23 Jul 2013 14:26:59	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	259,200	466,647	1.8003
23 Jul 2013 14:26:59	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	233,280	420,780	1.8038
23 Jul 2013 14:26:59	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	207,360	373,974	1.8035
11 Jul 2013 05:45:14	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	181,440	327,667	1.8059
10 Jul 2013 19:19:02	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	155,520	282,874	1.8189
09 Jul 2013 01:29:38	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	129,600	237,046	1.8291
07 Jul 2013 17:43:26	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	103,680	189,299	1.8258
06 Jul 2013 17:50:08	1129647	15880702	hadcm3n_n1hp_1920_40_008389774_2	77,760	141,733	1.8227