Task 12014977

Name	hadcm3igeo_w1s6_2000_80_06760747_2
Workunit	6964063
Created	12 Nov 2010, 15:31:12 UTC
Sent	15 Nov 2010, 7:57:50 UTC
Report deadline	16 May 2012, 4:40:59 UTC
Received	29 Jan 2011, 11:38:38 UTC
Server state	Over
Outcome	Computation error
Client state	Done
Exit status	22 (0x00000016) Unknown error code
Computer ID	384958
Run time
CPU time	39 days 21 hours 21 min 48 sec
Validate state	Invalid
Credit	2,612.74
Device peak FLOPS	1.51 GFLOPS
Application version	HadCM3L Coupled Model Experiment Optimised File I/O v6.04 windows_intelx86
Stderr	<core_client_version>5.2.13</core_client_version> <message>The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> scan: 0s5n_26_2000_atmos.gz scan: 0s5n_26_2000_ocean.gz scan: 971_flux_corr.anc.gz scan: boinc_lockfile scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: fogwhdck_0208_nickfluxcorr.anc.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w1s6_2000_80_06760747.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_2.gz scan: 0s5n_26_2000_atmos.gz scan: 0s5n_26_2000_ocean.gz scan: 971_flux_corr.anc.gz scan: boinc_lockfile scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: fogwhdck_0208_nickfluxcorr.anc.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w1s6_2000_80_06760747.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_2.gz scan: 0s5n_26_2000_atmos.gz scan: 0s5n_26_2000_ocean.gz scan: 971_flux_corr.anc.gz scan: boinc_lockfile scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: fogwhdck_0208_nickfluxcorr.anc.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w1s6_2000_80_06760747.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_2.gz CPDN Monitor - Quit request from BOINC... No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error ferror - Unit 21 - Return code = 32 Model crashed: Model crashed: Can not open stash_sets file No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/DMSNH3SO2A1B_19182082N cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/ozone_hadcm3_1919_2082 No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/DMSNH3SO2A1B_19182082N cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/SULPC_OXIDANTS_19_A2_1990 cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/ozone_hadcm3_1919_2082 BUFFIN: C I/O Error ferror - Unit 30 - Return code = 32 Model crashed: No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/DMSNH3SO2A1B_19182082N cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/SULPC_OXIDANTS_19_A2_1990 cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/ozone_hadcm3_1919_2082 BUFFIN: C I/O Error ferror - Unit 30 - Return code = 32 Model crashed: No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/DMSNH3SO2A1B_19182082N cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/SULPC_OXIDANTS_19_A2_1990 cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/ozone_hadcm3_1919_2082 BUFFIN: C I/O Error ferror - Unit 30 - Return code = 32 Model crashed: No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/DMSNH3SO2A1B_19182082N cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/SULPC_OXIDANTS_19_A2_1990 cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/ozone_hadcm3_1919_2082 BUFFIN: C I/O Error ferror - Unit 30 - Return code = 32 Model crashed: No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/DMSNH3SO2A1B_19182082N cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/SULPC_OXIDANTS_19_A2_1990 cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/ozone_hadcm3_1919_2082 BUFFIN: C I/O Error ferror - Unit 30 - Return code = 32 Model crashed: No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/DMSNH3SO2A1B_19182082N cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/SULPC_OXIDANTS_19_A2_1990 cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/ozone_hadcm3_1919_2082 BUFFIN: C I/O Error ferror - Unit 30 - Return code = 32 Model crashed: No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/DMSNH3SO2A1B_19182082N cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/SULPC_OXIDANTS_19_A2_1990 cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/ozone_hadcm3_1919_2082 BUFFIN: C I/O Error ferror - Unit 30 - Return code = 32 Model crashed: No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/DMSNH3SO2A1B_19182082N cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/ozone_hadcm3_1919_2082 No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/DMSNH3SO2A1B_19182082N cpdnmonitor: error reading file C:\Program Files\BOINC/projects/climateprediction.net/hadcm3igeo_w1s6_2000_80_06760747/datain/ancil/ozone_hadcm3_1919_2082 No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error ferror - Unit 116 - Return code = 32 Model crashed: No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error ferror - Unit 116 - Return code = 32 Model crashed: BUFFIN: C I/O Error ferror - Unit 116 - Return code = 32 Model crashed: CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error ferror - Unit 116 - Return code = 32 Model crashed: BUFFIN: C I/O Error ferror - Unit 116 - Return code = 32 Model crashed: BUFFIN: C I/O Error ferror - Unit 116 - Return code = 32 Model crashed: forrtl: Insufficient system resources exist to complete the requested service. CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3016, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Model crashed: (null) Model crashed: (null) Model crashed: Can not open stash_sets file No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error ferror - Unit 116 - Return code = 32 Model crashed: BUFFIN: C I/O Error ferror - Unit 116 - Return code = 32 Model crashed: BUFFIN: C I/O Error ferror - Unit 116 - Return code = 32 Model crashed: BUFFIN: C I/O Error ferror - Unit 116 - Return code = 32 Model crashed: BUFFIN: C I/O Error ferror - Unit 110 - Return code = 32 Model crashed: BUFFIN: C I/O Error ferror - Unit 110 - Return code = 32 Model crashed: Sorry, too many model crashes! :-( called boinc_finish </stderr_txt>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Nov 2010 09:16:28	384958	12014977	hadcm3igeo_w1s6_2000_80_06760747_2	207,360	643,247	3.1021
23 Nov 2010 06:36:55	384958	12014977	hadcm3igeo_w1s6_2000_80_06760747_2	181,440	563,285	3.1045
22 Nov 2010 06:02:03	384958	12014977	hadcm3igeo_w1s6_2000_80_06760747_2	155,520	486,768	3.1299
20 Nov 2010 22:53:58	384958	12014977	hadcm3igeo_w1s6_2000_80_06760747_2	129,600	400,861	3.0931
19 Nov 2010 20:02:10	384958	12014977	hadcm3igeo_w1s6_2000_80_06760747_2	103,680	319,801	3.0845
18 Nov 2010 16:43:23	384958	12014977	hadcm3igeo_w1s6_2000_80_06760747_2	77,760	238,932	3.0727
17 Nov 2010 14:21:51	384958	12014977	hadcm3igeo_w1s6_2000_80_06760747_2	51,840	159,602	3.0787
16 Nov 2010 12:15:10	384958	12014977	hadcm3igeo_w1s6_2000_80_06760747_2	25,920	80,397	3.1017