Task 13809843

Name	hadcm3n_ykwu_1940_40_007617526_1
Workunit	7795656
Created	22 Dec 2011, 22:46:36 UTC
Sent	22 Dec 2011, 22:53:23 UTC
Report deadline	23 Mar 2012, 6:20:34 UTC
Received	15 Jan 2012, 22:25:44 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1184704
Run time	5 days 21 hours 27 min 35 sec
CPU time	5 days 16 hours 22 min 18 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	2.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:36:41 (4644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:54:51 (6648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:54:55 (6648): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7000, iMonCtr=1 Model crash detected, will try to restart... 12:21:18 (7424): No heartbeat from core client for 30 sec - exiting 12:21:19 (7424): No heartbeat from core client for 30 sec - exiting 12:21:20 (7424): No heartbeat from core client for 30 sec - exiting 12:21:21 (7424): No heartbeat from core client for 30 sec - exiting 12:21:22 (7424): No heartbeat from core client for 30 sec - exiting 12:21:23 (7424): No heartbeat from core client for 30 sec - exiting 12:21:25 (7424): No heartbeat from core client for 30 sec - exiting 12:21:26 (7424): No heartbeat from core client for 30 sec - exiting 12:21:27 (7424): No heartbeat from core client for 30 sec - exiting 12:21:28 (7424): No heartbeat from core client for 30 sec - exiting 12:21:29 (7424): No heartbeat from core client for 30 sec - exiting 12:21:30 (7424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6592, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 13:06:58 (6796): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:02:04 (6664): No heartbeat from core client for 30 sec - exiting 14:02:05 (6664): No heartbeat from core client for 30 sec - exiting 14:02:06 (6664): No heartbeat from core client for 30 sec - exiting 14:02:08 (6664): No heartbeat from core client for 30 sec - exiting 14:02:09 (6664): No heartbeat from core client for 30 sec - exiting 14:02:10 (6664): No heartbeat from core client for 30 sec - exiting 14:02:11 (6664): No heartbeat from core client for 30 sec - exiting 14:02:12 (6664): No heartbeat from core client for 30 sec - exiting 14:02:13 (6664): No heartbeat from core client for 30 sec - exiting 14:02:14 (6664): No heartbeat from core client for 30 sec - exiting 14:02:15 (6664): No heartbeat from core client for 30 sec - exiting 14:02:16 (6664): No heartbeat from core client for 30 sec - exiting 14:02:17 (6664): No heartbeat from core client for 30 sec - exiting 14:02:18 (6664): No heartbeat from core client for 30 sec - exiting 14:02:20 (6664): No heartbeat from core client for 30 sec - exiting 14:02:21 (6664): No heartbeat from core client for 30 sec - exiting 14:02:22 (6664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:04:20 (5444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:04:21 (5444): No heartbeat from core client for 30 sec - exiting 14:04:22 (5444): No heartbeat from core client for 30 sec - exiting 14:04:23 (5444): No heartbeat from core client for 30 sec - exiting 14:04:24 (5444): No heartbeat from core client for 30 sec - exiting 14:04:25 (5444): No heartbeat from core client for 30 sec - exiting 14:04:26 (5444): No heartbeat from core client for 30 sec - exiting 14:04:28 (5444): No heartbeat from core client for 30 sec - exiting 14:04:29 (5444): No heartbeat from core client for 30 sec - exiting 14:04:30 (5444): No heartbeat from core client for 30 sec - exiting 14:04:31 (5444): No heartbeat from core client for 30 sec - exiting 14:05:59 (8036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:18:58 (6544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:19:00 (6544): No heartbeat from core client for 30 sec - exiting 14:19:01 (6544): No heartbeat from core client for 30 sec - exiting 14:19:02 (6544): No heartbeat from core client for 30 sec - exiting 14:19:03 (6544): No heartbeat from core client for 30 sec - exiting 14:19:04 (6544): No heartbeat from core client for 30 sec - exiting 14:19:05 (6544): No heartbeat from core client for 30 sec - exiting 14:19:06 (6544): No heartbeat from core client for 30 sec - exiting 14:19:07 (6544): No heartbeat from core client for 30 sec - exiting 14:19:08 (6544): No heartbeat from core client for 30 sec - exiting 14:19:10 (6544): No heartbeat from core client for 30 sec - exiting 15:14:01 (6280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts 16:51:51 (6128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:51:52 (6128): No heartbeat from core client for 30 sec - exiting 16:51:53 (6128): No heartbeat from core client for 30 sec - exiting cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts 16:53:08 (7424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7092, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7092, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7092, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7092, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts 16:58:35 (7092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts 17:00:25 (8940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts 17:02:39 (10172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts 17:03:52 (4740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts 17:04:38 (8728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts 17:06:42 (10420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:06:43 (10420): No heartbeat from core client for 30 sec - exiting 17:06:44 (10420): No heartbeat from core client for 30 sec - exiting 17:06:45 (10420): No heartbeat from core client for 30 sec - exiting 17:06:46 (10420): No heartbeat from core client for 30 sec - exiting 17:06:47 (10420): No heartbeat from core client for 30 sec - exiting 17:06:48 (10420): No heartbeat from core client for 30 sec - exiting 17:06:49 (10420): No heartbeat from core client for 30 sec - exiting 17:06:50 (10420): No heartbeat from core client for 30 sec - exiting 17:06:51 (10420): No heartbeat from core client for 30 sec - exiting 17:06:52 (10420): No heartbeat from core client for 30 sec - exiting cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6848, iMonCtr=1 Model crash detected, will try to restart... 17:10:29 (6848): No heartbeat from core client for 30 sec - exiting 17:10:30 (6848): No heartbeat from core client for 30 sec - exiting 17:10:31 (6848): No heartbeat from core client for 30 sec - exiting 17:10:32 (6848): No heartbeat from core client for 30 sec - exiting 17:10:33 (6848): No heartbeat from core client for 30 sec - exiting 17:10:34 (6848): No heartbeat from core client for 30 sec - exiting 17:10:35 (6848): No heartbeat from core client for 30 sec - exiting 17:10:36 (6848): No heartbeat from core client for 30 sec - exiting 17:10:37 (6848): No heartbeat from core client for 30 sec - exiting 17:10:38 (6848): No heartbeat from core client for 30 sec - exiting cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts 17:10:39 (6848): No heartbeat from core client for 30 sec - exiting cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ykwu_1940_40_007617526/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9892, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Jan 2012 03:13:07	1184704	13809843	hadcm3n_ykwu_1940_40_007617526_1	155,520	441,812	2.8409
01 Jan 2012 22:47:24	1184704	13809843	hadcm3n_ykwu_1940_40_007617526_1	129,600	362,917	2.8003
31 Dec 2011 22:29:57	1184704	13809843	hadcm3n_ykwu_1940_40_007617526_1	103,680	279,608	2.6968
30 Dec 2011 23:08:43	1184704	13809843	hadcm3n_ykwu_1940_40_007617526_1	77,760	198,768	2.5562
30 Dec 2011 22:08:19	1184704	13809843	hadcm3n_ykwu_1940_40_007617526_1	51,840	160,530	3.0966
28 Dec 2011 01:58:20	1184704	13809843	hadcm3n_ykwu_1940_40_007617526_1	25,920	91,160	3.5170