Task 12739616

Name	hadcm3n_o2zu_1900_40_007199213_1
Workunit	7397493
Created	28 Mar 2011, 14:05:25 UTC
Sent	31 Mar 2011, 16:27:15 UTC
Report deadline	30 Jun 2011, 23:54:26 UTC
Received	12 Apr 2011, 0:33:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1079363
Run time	9 days 22 hours 26 min 19 sec
CPU time	8 days 11 hours 59 min 5 sec
Validate state	Invalid
Credit	4,043.52
Device peak FLOPS	2.41 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/AvantGarde-Demi.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/Bookman-Demi.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/Courier-BoldOblique.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/Times-BoldItalic.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/Helvetica-Bold.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/Times-Roman.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/Times-Italic.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/NewCenturySchlbk-Bold.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/Bookman-LightItalic.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/globe.jpg 21:04:07 (3592): No heartbeat from core client for 30 sec - exiting 21:04:08 (3592): No heartbeat from core client for 30 sec - exiting 21:04:09 (3592): No heartbeat from core client for 30 sec - exiting 21:04:10 (3592): No heartbeat from core client for 30 sec - exiting 21:04:11 (3592): No heartbeat from core client for 30 sec - exiting 21:04:12 (3592): No heartbeat from core client for 30 sec - exiting 21:04:13 (3592): No heartbeat from core client for 30 sec - exiting 21:04:14 (3592): No heartbeat from core client for 30 sec - exiting 21:04:16 (3592): No heartbeat from core client for 30 sec - exiting 21:04:17 (3592): No heartbeat from core client for 30 sec - exiting 21:04:18 (3592): No heartbeat from core client for 30 sec - exiting 21:04:19 (3592): No heartbeat from core client for 30 sec - exiting 21:04:20 (3592): No heartbeat from core client for 30 sec - exiting 21:04:21 (3592): No heartbeat from core client for 30 sec - exiting 21:04:22 (3592): No heartbeat from core client for 30 sec - exiting 21:04:23 (3592): No heartbeat from core client for 30 sec - exiting 21:04:24 (3592): No heartbeat from core client for 30 sec - exiting 21:04:25 (3592): No heartbeat from core client for 30 sec - exiting 21:04:26 (3592): No heartbeat from core client for 30 sec - exiting 21:04:27 (3592): No heartbeat from core client for 30 sec - exiting 21:04:28 (3592): No heartbeat from core client for 30 sec - exiting 21:04:29 (3592): No heartbeat from core client for 30 sec - exiting 21:04:30 (3592): No heartbeat from core client for 30 sec - exiting 21:04:31 (3592): No heartbeat from core client for 30 sec - exiting 21:04:33 (3592): No heartbeat from core client for 30 sec - exiting 21:04:34 (3592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:04:35 (3592): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4412, iMonCtr=1 Model crash detected, will try to restart... 08:15:33 (3796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4000, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4696, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4768, iMonCtr=1 Model crash detected, will try to restart... 13:30:25 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4576, iMonCtr=1 Model crash detected, will try to restart... 16:18:25 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:19:26 (1888): Can't acquire lockfile (32) - waiting 35s Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3212, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4424, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:00:38 (4844): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3924, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3924, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4200, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4200, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4200, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4200, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Apr 2011 09:06:08	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	336,960	699,606	2.0762
12 Apr 2011 09:05:19	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	311,040	644,409	2.0718
12 Apr 2011 09:04:38	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	285,120	589,914	2.0690
08 Apr 2011 13:59:58	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	259,200	535,351	2.0654
07 Apr 2011 19:31:13	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	233,280	481,183	2.0627
07 Apr 2011 00:56:52	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	207,360	427,222	2.0603
06 Apr 2011 05:29:51	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	181,440	373,282	2.0573
05 Apr 2011 06:08:23	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	155,520	319,671	2.0555
04 Apr 2011 11:02:45	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	129,600	266,031	2.0527
03 Apr 2011 17:15:29	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	103,680	212,618	2.0507
03 Apr 2011 00:10:43	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	77,760	159,822	2.0553
02 Apr 2011 06:24:47	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	51,840	106,374	2.0520
01 Apr 2011 12:01:59	1079363	12739616	hadcm3n_o2zu_1900_40_007199213_1	25,920	52,796	2.0369