Task 15680553

Name	hadcm3n_3edb_1940_40_008266389_1
Workunit	8421513
Created	23 Mar 2013, 21:39:50 UTC
Sent	23 Mar 2013, 21:39:55 UTC
Report deadline	23 Jun 2013, 5:07:06 UTC
Received	26 Apr 2013, 21:54:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1256808
Run time	24 days 21 hours 51 min 17 sec
CPU time	15 days 23 hours 51 min 16 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	1.26 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:14:01 (2188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:14:02 (2188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 17:31:37 (3144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:38 (3144): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6320, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6320, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6320, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6320, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6320, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6320, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/ocean_restart.day after 11 attempts Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/ocean_restart.day after 11 attempts Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2224, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2224, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2224, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/ocean_restart.day after 11 attempts forrtl: severe (24): end-of-file during read, unit 5, file C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3n_3edb_1940_40_008266389\jobs\climate.cpdc Image PC Routine Line Source hadcm3n_um_6.07_w 007D9D2A Unknown Unknown Unknown hadcm3n_um_6.07_w 007D7058 Unknown Unknown Unknown hadcm3n_um_6.07_w 00780B60 Unknown Unknown Unknown hadcm3n_um_6.07_w 0077FD3A Unknown Unknown Unknown hadcm3n_um_6.07_w 007638F9 Unknown Unknown Unknown hadcm3n_um_6.07_w 006F38DC Unknown Unknown Unknown hadcm3n_um_6.07_w 006F2F48 Unknown Unknown Unknown hadcm3n_um_6.07_w 004083F3 Unknown Unknown Unknown hadcm3n_um_6.07_w 00408130 Unknown Unknown Unknown kernel32.dll 7C81776F Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2224, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/ocean_restart.day after 11 attempts forrtl: severe (24): end-of-file during read, unit 5, file C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3n_3edb_1940_40_008266389\jobs\climate.cpdc Image PC Routine Line Source hadcm3n_um_6.07_w 007D9D2A Unknown Unknown Unknown hadcm3n_um_6.07_w 007D7058 Unknown Unknown Unknown hadcm3n_um_6.07_w 00780B60 Unknown Unknown Unknown hadcm3n_um_6.07_w 0077FD3A Unknown Unknown Unknown hadcm3n_um_6.07_w 007638F9 Unknown Unknown Unknown hadcm3n_um_6.07_w 006F38DC Unknown Unknown Unknown hadcm3n_um_6.07_w 006F2F48 Unknown Unknown Unknown hadcm3n_um_6.07_w 004083F3 Unknown Unknown Unknown hadcm3n_um_6.07_w 00408130 Unknown Unknown Unknown kernel32.dll 7C81776F Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2224, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/ocean_restart.day after 11 attempts CPDN Monitor - Quit request from BOINC... cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/ocean_restart.day after 11 attempts Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\Documents and Settings\All Users\Application Data\BOINC/projects/climateprediction.net/hadcm3n_3edb_1940_40_008266389/dataout/ocean_restart.day after 11 attempts Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3092, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Apr 2013 09:02:06	1256808	15680553	hadcm3n_3edb_1940_40_008266389_1	285,120	1,357,605	4.7615
23 Apr 2013 00:26:16	1256808	15680553	hadcm3n_3edb_1940_40_008266389_1	259,200	1,234,121	4.7613
20 Apr 2013 14:15:28	1256808	15680553	hadcm3n_3edb_1940_40_008266389_1	233,280	1,109,995	4.7582
17 Apr 2013 01:04:17	1256808	15680553	hadcm3n_3edb_1940_40_008266389_1	207,360	985,994	4.7550
14 Apr 2013 18:43:10	1256808	15680553	hadcm3n_3edb_1940_40_008266389_1	181,440	861,596	4.7487
12 Apr 2013 11:10:11	1256808	15680553	hadcm3n_3edb_1940_40_008266389_1	155,520	738,526	4.7488
10 Apr 2013 05:21:57	1256808	15680553	hadcm3n_3edb_1940_40_008266389_1	129,600	614,479	4.7414
07 Apr 2013 17:26:29	1256808	15680553	hadcm3n_3edb_1940_40_008266389_1	103,680	491,544	4.7410
05 Apr 2013 12:26:20	1256808	15680553	hadcm3n_3edb_1940_40_008266389_1	77,760	368,389	4.7375
03 Apr 2013 07:09:50	1256808	15680553	hadcm3n_3edb_1940_40_008266389_1	51,840	245,540	4.7365
01 Apr 2013 01:21:53	1256808	15680553	hadcm3n_3edb_1940_40_008266389_1	25,920	123,352	4.7590