Task 12992994

Name	hadcm3n_s7pf_1940_40_007300995_1
Workunit	7498419
Created	21 Jun 2011, 1:49:22 UTC
Sent	21 Jun 2011, 1:49:35 UTC
Report deadline	20 Sep 2011, 9:16:46 UTC
Received	18 Jul 2011, 4:18:45 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1087383
Run time	16 days 11 hours 3 min 36 sec
CPU time	9 days 16 hours 57 min 28 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	1.35 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/s7pfko.pje2c10 Error converting file to netcdf: dataout/s7pfko.pie2c10 Error converting file to netcdf: dataout/s7pfko.pfe2c10 Error converting file to netcdf: dataout/s7pfka.phe2c10 Error converting file to netcdf: dataout/s7pfka.pge2c10 Error converting file to netcdf: dataout/s7pfka.pee2c10 Error converting file to netcdf: dataout/s7pfka.pde2c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:53:19 (1828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:47:32 (1288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:47:33 (1288): No heartbeat from core client for 30 sec - exiting 03:47:34 (1288): No heartbeat from core client for 30 sec - exiting 03:47:35 (1288): No heartbeat from core client for 30 sec - exiting 03:47:36 (1288): No heartbeat from core client for 30 sec - exiting 03:47:37 (1288): No heartbeat from core client for 30 sec - exiting 03:47:38 (1288): No heartbeat from core client for 30 sec - exiting 03:47:39 (1288): No heartbeat from core client for 30 sec - exiting 03:47:40 (1288): No heartbeat from core client for 30 sec - exiting 03:47:41 (1288): No heartbeat from core client for 30 sec - exiting 03:47:42 (1288): No heartbeat from core client for 30 sec - exiting 04:30:14 (4324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:30:15 (4324): No heartbeat from core client for 30 sec - exiting 04:30:16 (4324): No heartbeat from core client for 30 sec - exiting 04:30:17 (4324): No heartbeat from core client for 30 sec - exiting 04:30:18 (4324): No heartbeat from core client for 30 sec - exiting 04:30:19 (4324): No heartbeat from core client for 30 sec - exiting 04:30:20 (4324): No heartbeat from core client for 30 sec - exiting 04:30:21 (4324): No heartbeat from core client for 30 sec - exiting 04:30:22 (4324): No heartbeat from core client for 30 sec - exiting 04:30:23 (4324): No heartbeat from core client for 30 sec - exiting 04:30:24 (4324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:56:38 (1256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:56:40 (1256): No heartbeat from core client for 30 sec - exiting 03:56:41 (1256): No heartbeat from core client for 30 sec - exiting 03:56:42 (1256): No heartbeat from core client for 30 sec - exiting 03:56:43 (1256): No heartbeat from core client for 30 sec - exiting 03:56:44 (1256): No heartbeat from core client for 30 sec - exiting 03:56:45 (1256): No heartbeat from core client for 30 sec - exiting 03:56:46 (1256): No heartbeat from core client for 30 sec - exiting 03:56:47 (1256): No heartbeat from core client for 30 sec - exiting 03:56:48 (1256): No heartbeat from core client for 30 sec - exiting 03:56:49 (1256): No heartbeat from core client for 30 sec - exiting forrtl: The requested operation cannot be performed on a file with a user-mapped section open. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:13:49 (5000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:50 (5000): No heartbeat from core client for 30 sec - exiting 08:13:51 (5000): No heartbeat from core client for 30 sec - exiting 08:13:52 (5000): No heartbeat from core client for 30 sec - exiting 08:13:53 (5000): No heartbeat from core client for 30 sec - exiting 08:13:54 (5000): No heartbeat from core client for 30 sec - exiting 08:13:55 (5000): No heartbeat from core client for 30 sec - exiting 08:13:56 (5000): No heartbeat from core client for 30 sec - exiting 08:13:57 (5000): No heartbeat from core client for 30 sec - exiting 08:13:58 (5000): No heartbeat from core client for 30 sec - exiting 08:13:59 (5000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:44:24 (3136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:57:22 (5552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN MoniNo Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4888, selfPID=4888, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:58:17 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:18 (3456): No heartbeat from core client for 30 sec - exiting 01:58:19 (3456): No heartbeat from core client for 30 sec - exiting 01:58:20 (3456): No heartbeat from core client for 30 sec - exiting 01:58:21 (3456): No heartbeat from core client for 30 sec - exiting 01:58:22 (3456): No heartbeat from core client for 30 sec - exiting 01:58:23 (3456): No heartbeat from core client for 30 sec - exiting 02:04:09 (6080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:04:20 (6080): No heartbeat from core client for 30 sec - exiting 02:04:21 (6080): No heartbeat from core client for 30 sec - exiting 02:04:22 (6080): No heartbeat from core client for 30 sec - exiting 02:04:23 (6080): No heartbeat from core client for 30 sec - exiting 02:04:24 (6080): No heartbeat from core client for 30 sec - exiting 02:09:11 (6080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:09:12 (6080): No heartbeat from core client for 30 sec - exiting 02:09:13 (6080): No heartbeat from core client for 30 sec - exiting 02:09:14 (6080): No heartbeat from core client for 30 sec - exiting 02:09:15 (6080): No heartbeat from core client for 30 sec - exiting 02:09:16 (6080): No heartbeat from core client for 30 sec - exiting 02:09:17 (6080): No heartbeat from core client for 30 sec - exiting 02:09:18 (6080): No heartbeat from core client for 30 sec - exiting 02:09:19 (6080): No heartbeat from core client for 30 sec - exiting Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 04:09:17 (5348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:09:21 (5348): No heartbeat from core client for 30 sec - exiting 04:09:22 (5348): No heartbeat from core client for 30 sec - exiting 04:09:24 (5348): No heartbeat from core client for 30 sec - exiting Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 04:19:39 (2248): No heartbeat from core client for 30 sec - exiting 04:19:40 (2248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 16:31:13	1087383	12992994	hadcm3n_s7pf_1940_40_007300995_1	233,280	833,735	3.5740
09 Jul 2011 19:41:23	1087383	12992994	hadcm3n_s7pf_1940_40_007300995_1	207,360	742,965	3.5830
07 Jul 2011 15:43:00	1087383	12992994	hadcm3n_s7pf_1940_40_007300995_1	181,440	654,164	3.6054
07 Jul 2011 15:36:37	1087383	12992994	hadcm3n_s7pf_1940_40_007300995_1	155,520	564,927	3.6325
04 Jul 2011 08:33:05	1087383	12992994	hadcm3n_s7pf_1940_40_007300995_1	129,600	471,711	3.6397
02 Jul 2011 05:28:42	1087383	12992994	hadcm3n_s7pf_1940_40_007300995_1	103,680	374,394	3.6111
30 Jun 2011 15:09:31	1087383	12992994	hadcm3n_s7pf_1940_40_007300995_1	77,760	280,606	3.6086
25 Jun 2011 01:09:21	1087383	12992994	hadcm3n_s7pf_1940_40_007300995_1	51,840	186,026	3.5885
22 Jun 2011 21:53:02	1087383	12992994	hadcm3n_s7pf_1940_40_007300995_1	25,920	90,964	3.5094