Task 15781380

Name	hadcm3n_3gl4_1980_40_008367926_0
Workunit	8518785
Created	13 May 2013, 7:58:10 UTC
Sent	13 May 2013, 8:01:35 UTC
Report deadline	12 Aug 2013, 15:28:46 UTC
Received	22 Jun 2013, 4:06:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1195275
Run time	12 days 11 hours 55 min 7 sec
CPU time	12 days 4 hours 24 min 6 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	1.71 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 00:00:32 (5864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:01:16 (5564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:01:17 (5564): No heartbeat from core client for 30 sec - exiting 00:01:18 (5564): No heartbeat from core client for 30 sec - exiting 00:01:19 (5564): No heartbeat from core client for 30 sec - exiting 00:01:20 (5564): No heartbeat from core client for 30 sec - exiting 00:01:21 (5564): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:00:32 (5056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:00:34 (5056): No heartbeat from core client for 30 sec - exiting 23:00:35 (5056): No heartbeat from core client for 30 sec - exiting 23:00:36 (5056): No heartbeat from core client for 30 sec - exiting 23:00:37 (5056): No heartbeat from core client for 30 sec - exiting 23:00:38 (5056): No heartbeat from core client for 30 sec - exiting 23:00:39 (5056): No heartbeat from core client for 30 sec - exiting 23:00:40 (5056): No heartbeat from core client for 30 sec - exiting 23:00:41 (5056): No heartbeat from core client for 30 sec - exiting 23:00:42 (5056): No heartbeat from core client for 30 sec - exiting 23:00:43 (5056): No heartbeat from core client for 30 sec - exiting 23:00:44 (5056): No heartbeat from core client for 30 sec - exiting 23:01:40 (6352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 11:01:21 (6548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:00:48 (8112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:00:49 (8112): No heartbeat from core client for 30 sec - exiting 00:00:50 (8112): No heartbeat from core client for 30 sec - exiting 00:00:51 (8112): No heartbeat from core client for 30 sec - exiting 00:00:52 (8112): No heartbeat from core client for 30 sec - exiting 00:00:53 (8112): No heartbeat from core client for 30 sec - exiting 00:00:54 (8112): No heartbeat from core client for 30 sec - exiting 00:00:55 (8112): No heartbeat from core client for 30 sec - exiting 00:00:56 (8112): No heartbeat from core client for 30 sec - exiting 00:00:57 (8112): No heartbeat from core client for 30 sec - exiting 00:00:58 (8112): No heartbeat from core client for 30 sec - exiting 00:02:10 (5136): No heartbeat from core client for 30 sec - exiting 00:02:11 (5136): No heartbeat from core client for 30 sec - exiting 00:02:12 (5136): No heartbeat from core client for 30 sec - exiting 00:02:13 (5136): No heartbeat from core client for 30 sec - exiting 00:02:14 (5136): No heartbeat from core client for 30 sec - exiting 00:02:15 (5136): No heartbeat from core client for 30 sec - exiting 00:02:16 (5136): No heartbeat from core client for 30 sec - exiting 00:02:17 (5136): No heartbeat from core client for 30 sec - exiting 00:02:18 (5136): No heartbeat from core client for 30 sec - exiting 00:02:19 (5136): No heartbeat from core client for 30 sec - exiting 00:02:20 (5136): No heartbeat from core client for 30 sec - exiting 00:02:21 (5136): No heartbeat from core client for 30 sec - exiting 00:02:22 (5136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:54:40 (6300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:54:42 (6300): No heartbeat from core client for 30 sec - exiting 00:54:43 (6300): No heartbeat from core client for 30 sec - exiting 00:54:44 (6300): No heartbeat from core client for 30 sec - exiting 00:54:45 (6300): No heartbeat from core client for 30 sec - exiting 01:03:54 (8476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:35:33 (6224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:36:14 (9488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:01:50 (2604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:01:51 (2604): No heartbeat from core client for 30 sec - exiting 00:01:52 (2604): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 00:00:43 (4192): No heartbeat from core client for 30 sec - exiting 00:00:44 (4192): No heartbeat from core client for 30 sec - exiting 00:00:45 (4192): No heartbeat from core client for 30 sec - exiting 00:00:46 (4192): No heartbeat from core client for 30 sec - exiting 00:00:47 (4192): No heartbeat from core client for 30 sec - exiting 00:00:48 (4192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Jun 2013 11:14:03	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	440,640	1,021,624	2.3185
19 Jun 2013 07:15:41	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	414,720	983,545	2.3716
18 Jun 2013 04:16:45	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	388,800	944,177	2.4284
15 Jun 2013 00:51:22	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	362,880	887,525	2.4458
14 Jun 2013 14:31:12	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	336,960	851,049	2.5257
14 Jun 2013 02:19:30	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	311,040	812,801	2.6132
13 Jun 2013 14:41:30	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	285,120	774,804	2.7175
13 Jun 2013 04:07:04	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	259,200	737,036	2.8435
07 Jun 2013 10:51:22	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	233,280	671,857	2.8800
04 Jun 2013 12:01:18	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	207,360	596,575	2.8770
02 Jun 2013 06:02:45	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	181,440	520,032	2.8661
30 May 2013 05:07:47	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	155,520	444,183	2.8561
27 May 2013 07:23:29	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	129,600	368,870	2.8462
24 May 2013 09:43:11	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	103,680	293,671	2.8325
21 May 2013 09:13:12	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	77,760	218,935	2.8155
19 May 2013 05:36:05	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	51,840	144,299	2.7835
16 May 2013 09:07:26	1195275	15781380	hadcm3n_3gl4_1980_40_008367926_0	25,920	72,771	2.8075