Task 15512555

Name	hadcm3n_zdyn_1880_40_008252090_4
Workunit	8407214
Created	26 Dec 2012, 16:28:45 UTC
Sent	27 Dec 2012, 18:13:48 UTC
Report deadline	29 Mar 2013, 1:40:59 UTC
Received	1 Feb 2013, 16:07:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	987999
Run time	11 days 23 hours 30 min 29 sec
CPU time	11 days 23 hours 30 min 29 sec
Validate state	Invalid
Credit	6,842.88
Device peak FLOPS	2.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>5.10.30</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: The requested operation cannot be performed on a file with a user-mapped section open. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2580, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:58:06 (2728): No heartbeat from core client for 30 sec - exiting 02:58:07 (2728): No heartbeat from core client for 30 sec - exiting 02:58:08 (2728): No heartbeat from core client for 30 sec - exiting 02:58:09 (2728): No heartbeat from core client for 30 sec - exiting 02:58:10 (2728): No heartbeat from core client for 30 sec - exiting 02:58:11 (2728): No heartbeat from core client for 30 sec - exiting 02:58:12 (2728): No heartbeat from core client for 30 sec - exiting 02:58:13 (2728): No heartbeat from core client for 30 sec - exiting 02:58:14 (2728): No heartbeat from core client for 30 sec - exiting 02:58:15 (2728): No heartbeat from core client for 30 sec - exiting 02:58:17 (2728): No heartbeat from core client for 30 sec - exiting 02:58:18 (2728): No heartbeat from core client for 30 sec - exiting 02:58:19 (2728): No heartbeat from core client for 30 sec - exiting 02:58:20 (2728): No heartbeat from core client for 30 sec - exiting 02:58:21 (2728): No heartbeat from core client for 30 sec - exiting 02:58:22 (2728): No heartbeat from core client for 30 sec - exiting 02:58:23 (2728): No heartbeat from core client for 30 sec - exiting 02:58:24 (2728): No heartbeat from core client for 30 sec - exiting 02:58:25 (2728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:59:02 (2468): No heartbeat from core client for 30 sec - exiting 08:59:04 (2468): No heartbeat from core client for 30 sec - exiting 08:59:05 (2468): No heartbeat from core client for 30 sec - exiting 08:59:06 (2468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:06:20 (2460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Jan 2013 17:36:20	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	570,240	1,017,710	1.7847
31 Jan 2013 04:04:51	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	544,320	972,965	1.7875
30 Jan 2013 13:00:12	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	518,400	926,348	1.7869
29 Jan 2013 21:41:42	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	492,480	880,251	1.7874
29 Jan 2013 08:34:29	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	466,560	834,783	1.7892
28 Jan 2013 18:06:16	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	440,640	791,391	1.7960
27 Jan 2013 15:17:18	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	414,720	743,861	1.7936
26 Jan 2013 14:35:27	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	388,800	699,062	1.7980
25 Jan 2013 17:05:57	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	362,880	655,128	1.8054
24 Jan 2013 19:40:30	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	336,960	610,020	1.8104
23 Jan 2013 17:00:52	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	311,040	564,062	1.8135
22 Jan 2013 22:00:59	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	285,120	516,780	1.8125
21 Jan 2013 21:53:08	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	259,200	470,751	1.8162
20 Jan 2013 15:15:48	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	233,280	423,610	1.8159
19 Jan 2013 19:40:03	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	207,360	378,543	1.8255
18 Jan 2013 20:16:16	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	181,440	332,569	1.8329
18 Jan 2013 04:43:57	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	155,520	286,063	1.8394
17 Jan 2013 15:57:47	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	129,600	240,370	1.8547
17 Jan 2013 02:28:15	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	103,680	193,535	1.8667
16 Jan 2013 12:40:10	987999	15512555	hadcm3n_zdyn_1880_40_008252090_4	77,760	144,763	1.8617