Task 13011359

Name	hadcm3n_s3nr_1940_40_007301353_2
Workunit	7498777
Created	27 Jun 2011, 14:29:24 UTC
Sent	27 Jun 2011, 14:29:34 UTC
Report deadline	26 Sep 2011, 21:56:45 UTC
Received	15 Jul 2011, 16:53:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	864456
Run time	8 days 15 hours 57 min 10 sec
CPU time	8 days 6 hours 53 min 3 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.14 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:08:09 (5144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4876, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:20:33 (4452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/s3nrko.pje5c10 Error converting file to netcdf: dataout/s3nrko.pie5c10 Error converting file to netcdf: dataout/s3nrko.pfe5c10 Error converting file to netcdf: dataout/s3nrka.phe5c10 Error converting file to netcdf: dataout/s3nrka.pge5c10 Error converting file to netcdf: dataout/s3nrka.pee5c10 Error converting file to netcdf: dataout/s3nrka.pde5c10 Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4668, iMonCtr=1 Model crash detected, will try to restart... 00:03:18 (3904): No heartbeat from core client for 30 sec - exiting 00:03:19 (3904): No heartbeat from core client for 30 sec - exiting 00:03:20 (3904): No heartbeat from core client for 30 sec - exiting 00:03:21 (3904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:03:23 (3904): No heartbeat from core client for 30 sec - exiting 00:03:24 (3904): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/s3nrko.pje9c10 Error converting file to netcdf: dataout/s3nrko.pie9c10 Error converting file to netcdf: dataout/s3nrko.pfe9c10 Error converting file to netcdf: dataout/s3nrka.phe9c10 Error converting file to netcdf: dataout/s3nrka.pge9c10 Error converting file to netcdf: dataout/s3nrka.pee9c10 Error converting file to netcdf: dataout/s3nrka.pde9c10 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 14:22:31	864456	13011359	hadcm3n_s3nr_1940_40_007301353_2	233,280	716,002	3.0693
10 Jul 2011 17:48:28	864456	13011359	hadcm3n_s3nr_1940_40_007301353_2	207,360	633,203	3.0536
09 Jul 2011 15:27:37	864456	13011359	hadcm3n_s3nr_1940_40_007301353_2	181,440	549,886	3.0307
08 Jul 2011 14:40:01	864456	13011359	hadcm3n_s3nr_1940_40_007301353_2	155,520	466,405	2.9990
07 Jul 2011 15:42:07	864456	13011359	hadcm3n_s3nr_1940_40_007301353_2	129,600	386,556	2.9827
05 Jul 2011 17:51:52	864456	13011359	hadcm3n_s3nr_1940_40_007301353_2	103,680	309,628	2.9864
03 Jul 2011 21:25:45	864456	13011359	hadcm3n_s3nr_1940_40_007301353_2	77,760	231,606	2.9785
02 Jul 2011 22:49:33	864456	13011359	hadcm3n_s3nr_1940_40_007301353_2	51,840	153,454	2.9601
01 Jul 2011 22:39:15	864456	13011359	hadcm3n_s3nr_1940_40_007301353_2	25,920	77,082	2.9738