Task 13771701

Name	hadcm3n_ydtb_1980_40_007615322_2
Workunit	7793452
Created	13 Dec 2011, 1:27:44 UTC
Sent	13 Dec 2011, 1:31:50 UTC
Report deadline	13 Mar 2012, 8:59:01 UTC
Received	15 Jan 2012, 22:25:44 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1184704
Run time	11 days 22 hours 52 min 55 sec
CPU time	11 days 12 hours 46 min 39 sec
Validate state	Invalid
Credit	4,354.56
Device peak FLOPS	2.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:40:52 (1060): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16284, iMonCtr=1 Model crash detected, will try to restart... 11:17:37 (5476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:19:20 (7144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:29:32 (4772): No heartbeat from core client for 30 sec - exiting 11:29:33 (4772): No heartbeat from core client for 30 sec - exiting 11:29:34 (4772): No heartbeat from core client for 30 sec - exiting 11:29:35 (4772): No heartbeat from core client for 30 sec - exiting 11:29:37 (4772): No heartbeat from core client for 30 sec - exiting 11:29:38 (4772): No heartbeat from core client for 30 sec - exiting 11:29:39 (4772): No heartbeat from core client for 30 sec - exiting 11:29:40 (4772): No heartbeat from core client for 30 sec - exiting 11:29:41 (4772): No heartbeat from core client for 30 sec - exiting 11:29:42 (4772): No heartbeat from core client for 30 sec - exiting 11:29:43 (4772): No heartbeat from core client for 30 sec - exiting 11:29:44 (4772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:31:25 (8832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:34:00 (8144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:42 (7852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:54:51 (5236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:21:18 (5320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:21:21 (5320): No heartbeat from core client for 30 sec - exiting 12:21:22 (5320): No heartbeat from core client for 30 sec - exiting 12:21:23 (5320): No heartbeat from core client for 30 sec - exiting 12:21:24 (5320): No heartbeat from core client for 30 sec - exiting 12:39:11 (5716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:06:58 (5536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4860, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4860, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4860, iMonCtr=1 Model crash detected, will try to restart... 14:02:03 (5352): No heartbeat from core client for 30 sec - exiting 14:02:04 (5352): No heartbeat from core client for 30 sec - exiting 14:02:05 (5352): No heartbeat from core client for 30 sec - exiting 14:02:06 (5352): No heartbeat from core client for 30 sec - exiting 14:02:07 (5352): No heartbeat from core client for 30 sec - exiting 14:02:08 (5352): No heartbeat from core client for 30 sec - exiting 14:02:09 (5352): No heartbeat from core client for 30 sec - exiting 14:02:10 (5352): No heartbeat from core client for 30 sec - exiting 14:02:12 (5352): No heartbeat from core client for 30 sec - exiting 14:02:13 (5352): No heartbeat from core client for 30 sec - exiting 14:02:14 (5352): No heartbeat from core client for 30 sec - exiting 14:02:15 (5352): No heartbeat from core client for 30 sec - exiting 14:02:16 (5352): No heartbeat from core client for 30 sec - exiting 14:02:17 (5352): No heartbeat from core client for 30 sec - exiting 14:02:18 (5352): No heartbeat from core client for 30 sec - exiting 14:02:19 (5352): No heartbeat from core client for 30 sec - exiting 14:02:20 (5352): No heartbeat from core client for 30 sec - exiting 14:02:21 (5352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:04:19 (6068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:04:21 (6068): No heartbeat from core client for 30 sec - exiting 14:04:22 (6068): No heartbeat from core client for 30 sec - exiting 14:04:23 (6068): No heartbeat from core client for 30 sec - exiting 14:04:24 (6068): No heartbeat from core client for 30 sec - exiting 14:04:25 (6068): No heartbeat from core client for 30 sec - exiting 14:04:26 (6068): No heartbeat from core client for 30 sec - exiting 14:04:27 (6068): No heartbeat from core client for 30 sec - exiting 14:05:58 (2624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6676, iMonCtr=1 Model crash detected, will try to restart... 14:18:58 (5300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7016, iMonCtr=1 Model crash detected, will try to restart... 15:14:01 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1584, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts 16:53:05 (5440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6900, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6900, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6900, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts 16:58:35 (6900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts 17:00:24 (10180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts 17:02:31 (8308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts 17:03:52 (6176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts 17:04:32 (10132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts 17:06:32 (9884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts 17:07:43 (7892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts 17:08:25 (10456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:08:26 (10456): No heartbeat from core client for 30 sec - exiting 17:08:27 (10456): No heartbeat from core client for 30 sec - exiting 17:08:28 (10456): No heartbeat from core client for 30 sec - exiting 17:08:29 (10456): No heartbeat from core client for 30 sec - exiting cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts 17:10:27 (6912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:10:28 (6912): No heartbeat from core client for 30 sec - exiting 17:10:29 (6912): No heartbeat from core client for 30 sec - exiting 17:10:30 (6912): No heartbeat from core client for 30 sec - exiting 17:10:31 (6912): No heartbeat from core client for 30 sec - exiting 17:10:32 (6912): No heartbeat from core client for 30 sec - exiting 17:10:33 (6912): No heartbeat from core client for 30 sec - exiting 17:10:34 (6912): No heartbeat from core client for 30 sec - exiting 17:10:35 (6912): No heartbeat from core client for 30 sec - exiting 17:10:36 (6912): No heartbeat from core client for 30 sec - exiting cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6484, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6484, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydtb_1980_40_007615322/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6484, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Jan 2012 13:20:39	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	362,880	978,059	2.6953
02 Jan 2012 07:49:37	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	336,960	897,957	2.6649
01 Jan 2012 08:37:15	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	311,040	814,893	2.6199
31 Dec 2011 04:15:20	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	285,120	719,255	2.5226
30 Dec 2011 22:08:19	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	259,200	667,525	2.5753
28 Dec 2011 20:09:40	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	233,280	613,227	2.6287
27 Dec 2011 21:52:03	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	207,360	532,885	2.5699
26 Dec 2011 19:32:58	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	181,440	442,968	2.4414
25 Dec 2011 19:16:10	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	155,520	356,239	2.2906
24 Dec 2011 22:01:02	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	129,600	283,017	2.1838
24 Dec 2011 00:13:17	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	103,680	213,533	2.0595
23 Dec 2011 07:00:51	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	77,760	157,159	2.0211
22 Dec 2011 12:49:36	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	51,840	101,369	1.9554
13 Dec 2011 16:03:22	1184704	13771701	hadcm3n_ydtb_1980_40_007615322_2	25,920	48,453	1.8693