Task 13682418

Name	hadcm3n_yf3c_1940_40_007436701_2
Workunit	7634204
Created	2 Dec 2011, 4:31:25 UTC
Sent	2 Dec 2011, 4:32:55 UTC
Report deadline	2 Mar 2012, 12:00:06 UTC
Received	31 Dec 2011, 23:13:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1175367
Run time	6 days 1 hours 40 min 44 sec
CPU time	5 days 21 hours 48 min 20 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:03:36 (3728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:03:48 (3728): No heartbeat from core client for 30 sec - exiting 05:03:49 (3728): No heartbeat from core client for 30 sec - exiting 05:03:50 (3728): No heartbeat from core client for 30 sec - exiting 05:04:27 (4216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/yf3cko.pje1c10 Error converting file to netcdf: dataout/yf3cko.pie1c10 Error converting file to netcdf: dataout/yf3cko.pfe1c10 Error converting file to netcdf: dataout/yf3cka.phe1c10 Error converting file to netcdf: dataout/yf3cka.pge1c10 Error converting file to netcdf: dataout/yf3cka.pee1c10 Error converting file to netcdf: dataout/yf3cka.pde1c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/yf3cko.pje1c10 Error converting file to netcdf: dataout/yf3cko.pie1c10 Error converting file to netcdf: dataout/yf3cko.pfe1c10 Error converting file to netcdf: dataout/yf3cka.phe1c10 Error converting file to netcdf: dataout/yf3cka.pge1c10 Error converting file to netcdf: dataout/yf3cka.pee1c10 Error converting file to netcdf: dataout/yf3cka.pde1c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:49:17 (5736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:36:21 (3740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:36:22 (3740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 04:42:05 (4164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:42:16 (4164): No heartbeat from core client for 30 sec - exiting 04:42:17 (4164): No heartbeat from core client for 30 sec - exiting 04:42:18 (4164): No heartbeat from core client for 30 sec - exiting 04:42:19 (4164): No heartbeat from core client for 30 sec - exiting 04:42:20 (4164): No heartbeat from core client for 30 sec - exiting 04:42:21 (4164): No heartbeat from core client for 30 sec - exiting 04:42:22 (4164): No heartbeat from core client for 30 sec - exiting 04:42:23 (4164): No heartbeat from core client for 30 sec - exiting 04:42:24 (4164): No heartbeat from core client for 30 sec - exiting 04:42:25 (4164): No heartbeat from core client for 30 sec - exiting 04:43:06 (3624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:36:52 (1040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:03:57 (4668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:21:05 (2524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:10:07 (4960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:23:40 (1224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:23:19 (4496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:03:23 (2612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:04:03 (5264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/yf3cko.pje9c10 Error converting file to netcdf: dataout/yf3cko.pie9c10 Error converting file to netcdf: dataout/yf3cko.pfe9c10 Error converting file to netcdf: dataout/yf3cka.phe9c10 Error converting file to netcdf: dataout/yf3cka.pge9c10 Error converting file to netcdf: dataout/yf3cka.pee9c10 Error converting file to netcdf: dataout/yf3cka.pde9c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:43:47 (3516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:43:49 (3516): No heartbeat from core client for 30 sec - exiting 12:43:50 (3516): No heartbeat from core client for 30 sec - exiting 12:43:51 (3516): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 12:44:37 (3792): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_yf3c_1940_40_007436701/dataout/ocean_restart.day 17:21:20 (912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_yf3c_1940_40_007436701/dataout/ocean_restart.day BUFFIN: C I/O Error ferror - Unit 41 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 17:22:24 (4168): No heartbeat from core client for 30 sec - exiting 17:22:25 (4168): No heartbeat from core client for 30 sec - exiting 17:22:26 (4168): No heartbeat from core client for 30 sec - exiting 17:22:27 (4168): No heartbeat from core client for 30 sec - exiting 17:22:28 (4168): No heartbeat from core client for 30 sec - exiting 17:22:29 (4168): No heartbeat from core client for 30 sec - exiting 17:22:30 (4168): No heartbeat from core client for 30 sec - exiting 17:22:31 (4168): No heartbeat from core client for 30 sec - exiting 17:22:32 (4168): No heartbeat from core client for 30 sec - exiting 17:22:33 (4168): No heartbeat from core client for 30 sec - exiting 17:22:34 (4168): No heartbeat from core client for 30 sec - exiting 17:22:35 (4168): No heartbeat from core client for 30 sec - exiting 17:22:36 (4168): No heartbeat from core client for 30 sec - exiting 17:22:37 (4168): No heartbeat from core client for 30 sec - exiting 17:22:38 (4168): No heartbeat from core client for 30 sec - exiting 17:22:39 (4168): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_yf3c_1940_40_007436701/dataout/ocean_restart.day CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_yf3c_1940_40_007436701/dataout/ocean_restart.day BUFFIN: C I/O Error ferror - Unit 41 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_yf3c_1940_40_007436701/dataout/ocean_restart.day BUFFIN: C I/O Error ferror - Unit 41 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_yf3c_1940_40_007436701/dataout/ocean_restart.day BUFFIN: C I/O Error ferror - Unit 41 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_yf3c_1940_40_007436701/dataout/ocean_restart.day BUFFIN: C I/O Error ferror - Unit 41 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_yf3c_1940_40_007436701/dataout/ocean_restart.day BUFFIN: C I/O Error ferror - Unit 41 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Dec 2011 05:25:38	1175367	13682418	hadcm3n_yf3c_1940_40_007436701_2	233,280	461,890	1.9800
30 Dec 2011 09:06:13	1175367	13682418	hadcm3n_yf3c_1940_40_007436701_2	207,360	411,278	1.9834
29 Dec 2011 13:15:11	1175367	13682418	hadcm3n_yf3c_1940_40_007436701_2	181,440	360,184	1.9851
28 Dec 2011 22:20:38	1175367	13682418	hadcm3n_yf3c_1940_40_007436701_2	155,520	307,590	1.9778
28 Dec 2011 07:19:25	1175367	13682418	hadcm3n_yf3c_1940_40_007436701_2	129,600	255,244	1.9695
27 Dec 2011 16:34:15	1175367	13682418	hadcm3n_yf3c_1940_40_007436701_2	103,680	203,711	1.9648
27 Dec 2011 02:00:53	1175367	13682418	hadcm3n_yf3c_1940_40_007436701_2	77,760	152,821	1.9653
14 Dec 2011 01:35:47	1175367	13682418	hadcm3n_yf3c_1940_40_007436701_2	51,840	102,199	1.9714
13 Dec 2011 05:39:46	1175367	13682418	hadcm3n_yf3c_1940_40_007436701_2	25,920	51,163	1.9739