Task 16068087

Name	hadcm3n_o979_1900_40_008467048_1
Workunit	8617887
Created	15 Oct 2013, 17:14:47 UTC
Sent	15 Oct 2013, 17:14:52 UTC
Report deadline	15 Jan 2014, 0:42:03 UTC
Received	23 Oct 2013, 11:35:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1291838
Run time	6 days 3 hours 45 min 53 sec
CPU time	5 days 11 hours 52 min 57 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.34 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:33:45 (1656): No heartbeat from core client for 30 sec - exiting 07:33:46 (1656): No heartbeat from core client for 30 sec - exiting 07:33:47 (1656): No heartbeat from core client for 30 sec - exiting 07:33:48 (1656): No heartbeat from core client for 30 sec - exiting 07:33:49 (1656): No heartbeat from core client for 30 sec - exiting 07:33:50 (1656): No heartbeat from core client for 30 sec - exiting 07:33:51 (1656): No heartbeat from core client for 30 sec - exiting 07:33:52 (1656): No heartbeat from core client for 30 sec - exiting 07:33:53 (1656): No heartbeat from core client for 30 sec - exiting 07:33:54 (1656): No heartbeat from core client for 30 sec - exiting 07:33:56 (1656): No heartbeat from core client for 30 sec - exiting 07:33:57 (1656): No heartbeat from core client for 30 sec - exiting 07:33:58 (1656): No heartbeat from core client for 30 sec - exiting 07:33:59 (1656): No heartbeat from core client for 30 sec - exiting 07:34:00 (1656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6448, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6448, iMonCtr=1 Model crash detected, will try to restart... 18:28:41 (2796): No heartbeat from core client for 30 sec - exiting 18:28:42 (2796): No heartbeat from core client for 30 sec - exiting 18:28:43 (2796): No heartbeat from core client for 30 sec - exiting 18:28:44 (2796): No heartbeat from core client for 30 sec - exiting 18:28:45 (2796): No heartbeat from core client for 30 sec - exiting 18:28:46 (2796): No heartbeat from core client for 30 sec - exiting 18:28:47 (2796): No heartbeat from core client for 30 sec - exiting 18:28:48 (2796): No heartbeat from core client for 30 sec - exiting 18:28:49 (2796): No heartbeat from core client for 30 sec - exiting 18:28:51 (2796): No heartbeat from core client for 30 sec - exiting 18:28:52 (2796): No heartbeat from core client for 30 sec - exiting 18:28:53 (2796): No heartbeat from core client for 30 sec - exiting 18:28:54 (2796): No heartbeat from core client for 30 sec - exiting 18:28:55 (2796): No heartbeat from core client for 30 sec - exiting 18:28:56 (2796): No heartbeat from core client for 30 sec - exiting 18:28:57 (2796): No heartbeat from core client for 30 sec - exiting 18:28:58 (2796): No heartbeat from core client for 30 sec - exiting 18:28:59 (2796): No heartbeat from core client for 30 sec - exiting 18:29:00 (2796): No heartbeat from core client for 30 sec - exiting 18:29:02 (2796): No heartbeat from core client for 30 sec - exiting 18:29:03 (2796): No heartbeat from core client for 30 sec - exiting 18:29:04 (2796): No heartbeat from core client for 30 sec - exiting 18:29:05 (2796): No heartbeat from core client for 30 sec - exiting 18:29:06 (2796): No heartbeat from core client for 30 sec - exiting 18:29:07 (2796): No heartbeat from core client for 30 sec - exiting 18:29:08 (2796): No heartbeat from core client for 30 sec - exiting 18:29:09 (2796): No heartbeat from core client for 30 sec - exiting 18:29:10 (2796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4516, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4516, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4516, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4516, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Oct 2013 02:05:27	1291838	16068087	hadcm3n_o979_1900_40_008467048_1	233,280	450,725	1.9321
21 Oct 2013 11:04:06	1291838	16068087	hadcm3n_o979_1900_40_008467048_1	207,360	402,693	1.9420
20 Oct 2013 20:03:43	1291838	16068087	hadcm3n_o979_1900_40_008467048_1	181,440	353,380	1.9476
20 Oct 2013 05:37:50	1291838	16068087	hadcm3n_o979_1900_40_008467048_1	155,520	305,151	1.9621
19 Oct 2013 14:26:43	1291838	16068087	hadcm3n_o979_1900_40_008467048_1	129,600	256,036	1.9756
18 Oct 2013 10:05:41	1291838	16068087	hadcm3n_o979_1900_40_008467048_1	103,680	198,384	1.9134
17 Oct 2013 16:52:57	1291838	16068087	hadcm3n_o979_1900_40_008467048_1	77,760	147,946	1.9026
17 Oct 2013 00:14:02	1291838	16068087	hadcm3n_o979_1900_40_008467048_1	51,840	99,824	1.9256
16 Oct 2013 09:44:42	1291838	16068087	hadcm3n_o979_1900_40_008467048_1	25,920	49,743	1.9191