Task 15609904

Name	hadcm3n_zbvl_1880_40_008241933_4
Workunit	8397057
Created	16 Feb 2013, 3:54:51 UTC
Sent	16 Feb 2013, 3:55:09 UTC
Report deadline	18 May 2013, 11:22:20 UTC
Received	24 Mar 2013, 21:58:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1186467
Run time	8 days 8 hours 6 min 28 sec
CPU time	5 days 17 hours 39 min 24 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	2.91 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.52</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 00:49:42 (11044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:52:46 (8148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:38:54 (5564): No heartbeat from core client for 30 sec - exiting 17:38:55 (5564): No heartbeat from core client for 30 sec - exiting 17:38:56 (5564): No heartbeat from core client for 30 sec - exiting 17:38:57 (5564): No heartbeat from core client for 30 sec - exiting 17:38:58 (5564): No heartbeat from core client for 30 sec - exiting 17:38:59 (5564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:36:45 (8712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:28:35 (6032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:26:54 (9032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:18:48 (61072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:24:37 (61276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:48:57 (58424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:35:06 (62676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:42:34 (1336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:19:33 (33648): No heartbeat from core client for 30 sec - exiting 13:19:34 (33648): No heartbeat from core client for 30 sec - exiting 13:19:35 (33648): No heartbeat from core client for 30 sec - exiting 13:19:36 (33648): No heartbeat from core client for 30 sec - exiting 13:19:37 (33648): No heartbeat from core client for 30 sec - exiting 13:19:38 (33648): No heartbeat from core client for 30 sec - exiting 13:19:39 (33648): No heartbeat from core client for 30 sec - exiting 13:19:40 (33648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:56:58 (4084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:53:11 (39900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:21:29 (35780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:37:50 (5768): No heartbeat from core client for 30 sec - exiting 06:37:51 (5768): No heartbeat from core client for 30 sec - exiting 06:37:52 (5768): No heartbeat from core client for 30 sec - exiting 06:37:53 (5768): No heartbeat from core client for 30 sec - exiting 06:37:54 (5768): No heartbeat from core client for 30 sec - exiting 06:37:55 (5768): No heartbeat from core client for 30 sec - exiting 06:37:56 (5768): No heartbeat from core client for 30 sec - exiting 06:37:57 (5768): No heartbeat from core client for 30 sec - exiting 06:37:58 (5768): No heartbeat from core client for 30 sec - exiting 06:37:59 (5768): No heartbeat from core client for 30 sec - exiting 06:38:01 (5768): No heartbeat from core client for 30 sec - exiting 06:38:02 (5768): No heartbeat from core client for 30 sec - exiting 06:38:03 (5768): No heartbeat from core client for 30 sec - exiting 06:38:04 (5768): No heartbeat from core client for 30 sec - exiting 06:38:05 (5768): No heartbeat from core client for 30 sec - exiting 06:38:06 (5768): No heartbeat from core client for 30 sec - exiting 06:38:07 (5768): No heartbeat from core client for 30 sec - exiting 06:38:08 (5768): No heartbeat from core client for 30 sec - exiting 06:38:09 (5768): No heartbeat from core client for 30 sec - exiting 06:38:10 (5768): No heartbeat from core client for 30 sec - exiting 06:38:11 (5768): No heartbeat from core client for 30 sec - exiting 06:38:13 (5768): No heartbeat from core client for 30 sec - exiting 06:38:14 (5768): No heartbeat from core client for 30 sec - exiting 06:38:15 (5768): No heartbeat from core client for 30 sec - exiting 06:38:16 (5768): No heartbeat from core client for 30 sec - exiting 06:38:17 (5768): No heartbeat from core client for 30 sec - exiting 06:38:18 (5768): No heartbeat from core client for 30 sec - exiting 06:38:19 (5768): No heartbeat from core client for 30 sec - exiting 06:38:20 (5768): No heartbeat from core client for 30 sec - exiting 06:38:21 (5768): No heartbeat from core client for 30 sec - exiting 06:38:22 (5768): No heartbeat from core client for 30 sec - exiting 06:38:23 (5768): No heartbeat from core client for 30 sec - exiting 06:38:25 (5768): No heartbeat from core client for 30 sec - exiting 06:38:26 (5768): No heartbeat from core client for 30 sec - exiting 06:38:27 (5768): No heartbeat from core client for 30 sec - exiting 06:38:28 (5768): No heartbeat from core client for 30 sec - exiting 06:38:29 (5768): No heartbeat from core client for 30 sec - exiting 06:38:30 (5768): No heartbeat from core client for 30 sec - exiting 06:38:31 (5768): No heartbeat from core client for 30 sec - exiting 06:38:32 (5768): No heartbeat from core client for 30 sec - exiting 06:38:33 (5768): No heartbeat from core client for 30 sec - exiting 06:38:34 (5768): No heartbeat from core client for 30 sec - exiting 06:38:36 (5768): No heartbeat from core client for 30 sec - exiting 06:38:37 (5768): No heartbeat from core client for 30 sec - exiting 06:38:38 (5768): No heartbeat from core client for 30 sec - exiting 06:38:39 (5768): No heartbeat from core client for 30 sec - exiting 06:38:40 (5768): No heartbeat from core client for 30 sec - exiting 06:38:41 (5768): No heartbeat from core client for 30 sec - exiting 06:38:42 (5768): No heartbeat from core client for 30 sec - exiting 06:38:43 (5768): No heartbeat from core client for 30 sec - exiting 06:38:44 (5768): No heartbeat from core client for 30 sec - exiting 06:38:45 (5768): No heartbeat from core client for 30 sec - exiting 06:38:46 (5768): No heartbeat from core client for 30 sec - exiting 06:38:48 (5768): No heartbeat from core client for 30 sec - exiting 06:38:49 (5768): No heartbeat from core client for 30 sec - exiting 06:38:50 (5768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:40:36 (7108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:41:46 (6964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:41:47 (6964): No heartbeat from core client for 30 sec - exiting 06:41:48 (6964): No heartbeat from core client for 30 sec - exiting 06:41:49 (6964): No heartbeat from core client for 30 sec - exiting 06:41:50 (6964): No heartbeat from core client for 30 sec - exiting 06:41:51 (6964): No heartbeat from core client for 30 sec - exiting 06:41:53 (6964): No heartbeat from core client for 30 sec - exiting 06:47:08 (512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:01:15 (11128): No heartbeat from core client for 30 sec - exiting 22:01:16 (11128): No heartbeat from core client for 30 sec - exiting 22:01:17 (11128): No heartbeat from core client for 30 sec - exiting 22:01:18 (11128): No heartbeat from core client for 30 sec - exiting 22:01:19 (11128): No heartbeat from core client for 30 sec - exiting 22:01:20 (11128): No heartbeat from core client for 30 sec - exiting 22:01:21 (11128): No heartbeat from core client for 30 sec - exiting 22:01:22 (11128): No heartbeat from core client for 30 sec - exiting 22:01:23 (11128): No heartbeat from core client for 30 sec - exiting 22:01:24 (11128): No heartbeat from core client for 30 sec - exiting 22:01:25 (11128): No heartbeat from core client for 30 sec - exiting 22:01:26 (11128): No heartbeat from core client for 30 sec - exiting 22:01:27 (11128): No heartbeat from core client for 30 sec - exiting 22:01:28 (11128): No heartbeat from core client for 30 sec - exiting 22:01:29 (11128): No heartbeat from core client for 30 sec - exiting 22:01:30 (11128): No heartbeat from core client for 30 sec - exiting 22:01:31 (11128): No heartbeat from core client for 30 sec - exiting 22:01:32 (11128): No heartbeat from core client for 30 sec - exiting 22:01:33 (11128): No heartbeat from core client for 30 sec - exiting 22:01:34 (11128): No heartbeat from core client for 30 sec - exiting 22:01:35 (11128): No heartbeat from core client for 30 sec - exiting 22:01:36 (11128): No heartbeat from core client for 30 sec - exiting 22:01:37 (11128): No heartbeat from core client for 30 sec - exiting 22:01:38 (11128): No heartbeat from core client for 30 sec - exiting 22:01:39 (11128): No heartbeat from core client for 30 sec - exiting 22:01:40 (11128): No heartbeat from core client for 30 sec - exiting 22:01:41 (11128): No heartbeat from core client for 30 sec - exiting 22:01:42 (11128): No heartbeat from core client for 30 sec - exiting 22:01:43 (11128): No heartbeat from core client for 30 sec - exiting 22:01:44 (11128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:39:17 (29916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:26:12 (32832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:44:15 (30048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:30:36 (14368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:14:14 (2780): No heartbeat from core client for 30 sec - exiting 08:14:15 (2780): No heartbeat from core client for 30 sec - exiting 08:14:16 (2780): No heartbeat from core client for 30 sec - exiting 08:14:17 (2780): No heartbeat from core client for 30 sec - exiting 08:14:18 (2780): No heartbeat from core client for 30 sec - exiting 08:14:19 (2780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:31:04 (2428): No heartbeat from core client for 30 sec - exiting 20:31:05 (2428): No heartbeat from core client for 30 sec - exiting 20:31:06 (2428): No heartbeat from core client for 30 sec - exiting 20:31:07 (2428): No heartbeat from core client for 30 sec - exiting 20:31:08 (2428): No heartbeat from core client for 30 sec - exiting 20:31:09 (2428): No heartbeat from core client for 30 sec - exiting 20:31:10 (2428): No heartbeat from core client for 30 sec - exiting 20:31:11 (2428): No heartbeat from core client for 30 sec - exiting 20:31:12 (2428): No heartbeat from core client for 30 sec - exiting 20:31:13 (2428): No heartbeat from core client for 30 sec - exiting 20:31:14 (2428): No heartbeat from core client for 30 sec - exiting 20:31:15 (2428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:00:48 (7148): No heartbeat from core client for 30 sec - exiting 15:00:49 (7148): No heartbeat from core client for 30 sec - exiting 15:00:50 (7148): No heartbeat from core client for 30 sec - exiting 15:00:51 (7148): No heartbeat from core client for 30 sec - exiting 15:00:52 (7148): No heartbeat from core client for 30 sec - exiting 15:00:53 (7148): No heartbeat from core client for 30 sec - exiting 15:00:54 (7148): No heartbeat from core client for 30 sec - exiting 15:01:28 (7148): No heartbeat from core client for 30 sec - exiting 15:01:29 (7148): No heartbeat from core client for 30 sec - exiting 15:01:30 (7148): No heartbeat from core client for 30 sec - exiting 15:01:32 (7148): No heartbeat from core client for 30 sec - exiting 15:01:33 (7148): No heartbeat from core client for 30 sec - exiting 15:01:34 (7148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:30:49 (24704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=44456, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=44456, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=44456, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=44456, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3708, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3708, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Mar 2013 07:24:27	1186467	15609904	hadcm3n_zbvl_1880_40_008241933_4	285,120	470,814	1.6513
12 Mar 2013 10:01:54	1186467	15609904	hadcm3n_zbvl_1880_40_008241933_4	259,200	427,309	1.6486
09 Mar 2013 16:55:44	1186467	15609904	hadcm3n_zbvl_1880_40_008241933_4	233,280	384,552	1.6485
07 Mar 2013 18:51:20	1186467	15609904	hadcm3n_zbvl_1880_40_008241933_4	207,360	342,189	1.6502
04 Mar 2013 06:02:38	1186467	15609904	hadcm3n_zbvl_1880_40_008241933_4	181,440	300,524	1.6563
02 Mar 2013 03:05:47	1186467	15609904	hadcm3n_zbvl_1880_40_008241933_4	155,520	257,909	1.6584
28 Feb 2013 16:20:39	1186467	15609904	hadcm3n_zbvl_1880_40_008241933_4	129,600	213,336	1.6461
25 Feb 2013 19:54:23	1186467	15609904	hadcm3n_zbvl_1880_40_008241933_4	103,680	170,824	1.6476
23 Feb 2013 22:10:43	1186467	15609904	hadcm3n_zbvl_1880_40_008241933_4	77,760	129,286	1.6626
21 Feb 2013 23:29:58	1186467	15609904	hadcm3n_zbvl_1880_40_008241933_4	51,840	85,570	1.6507
17 Feb 2013 22:45:56	1186467	15609904	hadcm3n_zbvl_1880_40_008241933_4	25,920	43,102	1.6629