Task 13847257

Name	hadcm3n_ylpp_1900_40_007513858_2
Workunit	7711333
Created	2 Jan 2012, 1:54:09 UTC
Sent	2 Jan 2012, 1:54:13 UTC
Report deadline	2 Apr 2012, 9:21:24 UTC
Received	24 Feb 2012, 3:24:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1057787
Run time	12 days 23 hours 35 min 7 sec
CPU time	6 days 4 hours 6 min 28 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	2.81 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 19:54:33 (1984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:55:04 (1668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:55:05 (1668): No heartbeat from core client for 30 sec - exiting 22:29:41 (3408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:42 (3408): No heartbeat from core client for 30 sec - exiting 01:52:49 (1988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:52:51 (1988): No heartbeat from core client for 30 sec - exiting 04:29:09 (1560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:29:11 (1560): No heartbeat from core client for 30 sec - exiting 04:29:14 (1560): No heartbeat from core client for 30 sec - exiting 06:05:38 (2688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:05:40 (2688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:29:31 (3640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:29:32 (3640): No heartbeat from core client for 30 sec - exiting 22:03:47 (2720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:03:49 (2720): No heartbeat from core client for 30 sec - exiting 22:08:05 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:08:07 (4712): No heartbeat from core client for 30 sec - exiting 23:28:35 (5172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:28:39 (5172): No heartbeat from core client for 30 sec - exiting 23:28:58 (5172): No heartbeat from core client for 30 sec - exiting 23:29:52 (5848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:29:54 (5848): No heartbeat from core client for 30 sec - exiting 23:30:50 (4152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:30:52 (4152): No heartbeat from core client for 30 sec - exiting 23:34:36 (1168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:34:38 (1168): No heartbeat from core client for 30 sec - exiting 23:35:41 (996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:35:42 (996): No heartbeat from core client for 30 sec - exiting 23:37:07 (4280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:37:09 (4280): No heartbeat from core client for 30 sec - exiting 23:38:04 (5432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:38:06 (5432): No heartbeat from core client for 30 sec - exiting 23:38:58 (5768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:39:00 (5768): No heartbeat from core client for 30 sec - exiting 23:40:04 (5884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:41:26 (5296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:41:28 (5296): No heartbeat from core client for 30 sec - exiting 23:43:04 (3040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:43:06 (3040): No heartbeat from core client for 30 sec - exiting 23:44:03 (4440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:44:04 (4440): No heartbeat from core client for 30 sec - exiting 23:49:44 (4320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:49:46 (4320): No heartbeat from core client for 30 sec - exiting 23:51:34 (5152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:51:37 (5152): No heartbeat from core client for 30 sec - exiting 23:52:58 (5584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:53:00 (5584): No heartbeat from core client for 30 sec - exiting 23:54:52 (5832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:54:54 (5832): No heartbeat from core client for 30 sec - exiting 23:56:58 (5240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:57:01 (5240): No heartbeat from core client for 30 sec - exiting 23:58:35 (480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:02:09 (4020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:02:10 (4020): No heartbeat from core client for 30 sec - exiting 00:03:11 (4924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:19 (1888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:20 (1888): No heartbeat from core client for 30 sec - exiting 03:41:11 (5612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:12 (5612): No heartbeat from core client for 30 sec - exiting 08:55:42 (4792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:44 (4792): No heartbeat from core client for 30 sec - exiting 10:20:18 (5788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:20:20 (5788): No heartbeat from core client for 30 sec - exiting 10:31:13 (1032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:31:14 (1032): No heartbeat from core client for 30 sec - exiting 18:06:41 (5824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:06:42 (5824): No heartbeat from core client for 30 sec - exiting 07:00:19 (4652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:00:20 (4652): No heartbeat from core client for 30 sec - exiting 07:00:22 (4652): No heartbeat from core client for 30 sec - exiting 11:27:52 (5916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:27:53 (5916): No heartbeat from core client for 30 sec - exiting 14:39:52 (3488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:49:56 (3076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:49:57 (3076): No heartbeat from core client for 30 sec - exiting 18:49:58 (3076): No heartbeat from core client for 30 sec - exiting 18:49:59 (3076): No heartbeat from core client for 30 sec - exiting 18:50:00 (3076): No heartbeat from core client for 30 sec - exiting 18:50:01 (3076): No heartbeat from core client for 30 sec - exiting 18:50:02 (3076): No heartbeat from core client for 30 sec - exiting 18:50:03 (3076): No heartbeat from core client for 30 sec - exiting 18:50:04 (3076): No heartbeat from core client for 30 sec - exiting 18:50:05 (3076): No heartbeat from core client for 30 sec - exiting 18:50:06 (3076): No heartbeat from core client for 30 sec - exiting 22:25:53 (5752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:25:54 (5752): No heartbeat from core client for 30 sec - exiting 22:25:56 (5752): No heartbeat from core client for 30 sec - exiting 02:48:19 (2316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:48:20 (2316): No heartbeat from core client for 30 sec - exiting 02:48:22 (2316): No heartbeat from core client for 30 sec - exiting 02:48:23 (2316): No heartbeat from core client for 30 sec - exiting 05:36:43 (3168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:36:44 (3168): No heartbeat from core client for 30 sec - exiting 10:15:32 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:15:33 (5504): No heartbeat from core client for 30 sec - exiting 10:15:34 (5504): No heartbeat from core client for 30 sec - exiting 11:39:29 (5436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:39:31 (5436):12:43:38 (5296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:43:40 (5296): No heartbeat from core client for 30 sec - exiting 12:43:41 (5296): No heartbeat from core client for 30 sec - exiting 17:09:56 (3232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:09:58 (3232): No heartbeat from core client for 30 sec - exiting 20:42:14 (5776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:42:15 (5776): No heartbeat from core client for 30 sec - exiting 20:42:16 (5776):21:21:49 (6028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:21:50 (6028): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 11:45:04 (4936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:45:06 (4936): No heartbeat from core client for 30 sec - exiting 11:45:07 (4936): No heartbeat from core client for 30 sec - exiting 11:45:08 (4936): No heartbeat from core client for 30 sec - exiting 11:45:09 (4936): No heartbeat from core client for 30 sec - exiting 11:45:10 (4936): No heartbeat from core client for 30 sec - exiting 11:45:11 (4936): No heartbeat from core client for 30 sec - exiting 12:17:23 (4340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:17:24 (4340): No heartbeat from core client for 30 sec - exiting 15:46:06 (3320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:46:07 (3320): No heartbeat from core client for 30 sec - exiting 15:46:09 (3320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 11:53:18 (1768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:53:20 (1768): No heartbeat from core client for 30 sec - exiting 21:44:08 (3984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:44:09 (3984): No heartbeat from core client for 30 sec - exiting 21:44:10 (3984): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 03:18:33 (676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:00:51 (3116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:00:52 (3116): No heartbeat from core client for 30 sec - exiting 12:00:53 (3116): No heartbeat from core client for 30 sec - exiting 12:00:54 (3116): No heartbeat from core client for 30 sec - exiting 12:00:56 (3116): No heartbeat from core client for 30 sec - exiting 12:00:57 (3116): No heartbeat from core client for 30 sec - exiting 12:00:58 (3116): No heartbeat from core client for 30 sec - exiting 12:00:59 (3116): No heartbeat from core client for 30 sec - exiting 12:01:00 (3116): No heartbeat from core client for 30 sec - exiting 12:01:01 (3116): No heartbeat from core client for 30 sec - exiting 12:46:26 (1640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:46:27 (1640): No heartbeat from core client for 30 sec - exiting 19:06:10 (3984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:06:11 (3984): No heartbeat from core client for 30 sec - exiting 19:06:13 (3984): No heartbeat from core client for 30 sec - exiting 19:32:39 (2812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:32:41 (2812): No heartbeat from core client for 30 sec - exiting 04:15:14 (3268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:40:19 (3796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4068, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4068, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4068, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4068, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4068, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4068, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Feb 2012 02:22:57	1057787	13847257	hadcm3n_ylpp_1900_40_007513858_2	285,120	522,567	1.8328
21 Feb 2012 20:36:45	1057787	13847257	hadcm3n_ylpp_1900_40_007513858_2	259,200	475,437	1.8342
20 Feb 2012 19:12:15	1057787	13847257	hadcm3n_ylpp_1900_40_007513858_2	233,280	427,094	1.8308
19 Feb 2012 21:14:32	1057787	13847257	hadcm3n_ylpp_1900_40_007513858_2	207,360	378,439	1.8250
19 Feb 2012 00:57:57	1057787	13847257	hadcm3n_ylpp_1900_40_007513858_2	181,440	329,570	1.8164
18 Feb 2012 02:34:33	1057787	13847257	hadcm3n_ylpp_1900_40_007513858_2	155,520	280,897	1.8062
16 Feb 2012 02:57:06	1057787	13847257	hadcm3n_ylpp_1900_40_007513858_2	129,600	232,745	1.7959
15 Feb 2012 05:37:31	1057787	13847257	hadcm3n_ylpp_1900_40_007513858_2	103,680	184,193	1.7766
14 Feb 2012 03:53:21	1057787	13847257	hadcm3n_ylpp_1900_40_007513858_2	77,760	136,093	1.7502
12 Feb 2012 18:22:56	1057787	13847257	hadcm3n_ylpp_1900_40_007513858_2	51,840	91,882	1.7724
10 Feb 2012 22:17:24	1057787	13847257	hadcm3n_ylpp_1900_40_007513858_2	25,920	45,625	1.7602