Task 12711959

Name	hadcm3igeo_w30e_2000_80_07192722_0
Workunit	7391003
Created	24 Mar 2011, 15:02:42 UTC
Sent	27 Mar 2011, 2:38:39 UTC
Report deadline	24 Sep 2012, 23:21:48 UTC
Received	18 Jul 2011, 9:39:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1116868
Run time	29 days 12 hours 4 min 54 sec
CPU time	27 days 16 hours 13 min 41 sec
Validate state	Invalid
Credit	15,349.82
Device peak FLOPS	2.74 GFLOPS
Application version	HadCM3L Coupled Model Experiment Optimised File I/O v6.04 windows_intelx86
Stderr	<core_client_version>6.12.26</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> scan: 0mct_05987932_atmos.day.gz scan: 0mct_05987932_ocean.day.gz scan: 1029_flux_corr.anc.gz scan: boinc_lockfile scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w30e_2000_80_07192722.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: heblhdck_0208_nickfluxcorr.anc.gz scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_3.gz BUFFIN: C I/O Error ferror - Unit 63 - Return code = 32 Model crashed: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6488, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6488, iMonCtr=1 Model crash detected, will try to restart... CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6488, iMonCtr=1 Model crash detected, will try to restart... No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... scan: 0mct_05987932_atmos.day.gz scan: 0mct_05987932_ocean.day.gz scan: 1029_flux_corr.anc.gz scan: boinc_lockfile scan: boinc_task_state.xml scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w30e_2000_80_07192722.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: heblhdck_0208_nickfluxcorr.anc.gz scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_3.gz Suspended CPDN Monitor - Quit request from BOINC... scan: 0mct_05987932_atmos.day.gz scan: 0mct_05987932_ocean.day.gz scan: 1029_flux_corr.anc.gz scan: boinc_lockfile scan: boinc_task_state.xml scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w30e_2000_80_07192722.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: heblhdck_0208_nickfluxcorr.anc.gz scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: stderrgfx.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_3.gz No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... scan: 0mct_05987932_atmos.day.gz scan: 0mct_05987932_ocean.day.gz scan: 1029_flux_corr.anc.gz scan: boinc_lockfile scan: boinc_task_state.xml scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w30e_2000_80_07192722.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: heblhdck_0208_nickfluxcorr.anc.gz scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: stderrgfx.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_3.gz scan: 0mct_05987932_atmos.day.gz scan: 0mct_05987932_ocean.day.gz scan: 1029_flux_corr.anc.gz scan: boinc_lockfile scan: boinc_task_state.xml scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w30e_2000_80_07192722.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: heblhdck_0208_nickfluxcorr.anc.gz scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: stderrgfx.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_3.gz CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12248, iMonCtr=1 Model crash detected, will try to restart... scan: 0mct_05987932_atmos.day.gz scan: 0mct_05987932_ocean.day.gz scan: 1029_flux_corr.anc.gz scan: boinc_lockfile scan: boinc_task_state.xml scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w30e_2000_80_07192722.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: heblhdck_0208_nickfluxcorr.anc.gz scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: stderrgfx.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_3.gz scan: 0mct_05987932_atmos.day.gz scan: 0mct_05987932_ocean.day.gz scan: 1029_flux_corr.anc.gz scan: boinc_lockfile scan: boinc_task_state.xml scan: boinc_ufs_cpdnout1.zip scan: boinc_ufs_cpdnout2.zip scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w30e_2000_80_07192722.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: heblhdck_0208_nickfluxcorr.anc.gz scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: stderrgfx.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_3.gz No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting scan: 0mct_05987932_atmos.day.gz scan: 0mct_05987932_ocean.day.gz scan: 1029_flux_corr.anc.gz scan: boinc_lockfile scan: boinc_task_state.xml scan: boinc_ufs_cpdnout1.zip scan: boinc_ufs_cpdnout2.zip scan: boinc_ufs_cpdnout3.zip scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w30e_2000_80_07192722.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: heblhdck_0208_nickfluxcorr.anc.gz scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: stderrgfx.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_3.gz No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... scan: 0mct_05987932_atmos.day.gz scan: 0mct_05987932_ocean.day.gz scan: 1029_flux_corr.anc.gz scan: boinc_lockfile scan: boinc_task_state.xml scan: boinc_ufs_cpdnout1.zip scan: boinc_ufs_cpdnout2.zip scan: boinc_ufs_cpdnout3.zip scan: boinc_ufs_cpdnout4.zip scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w30e_2000_80_07192722.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: heblhdck_0208_nickfluxcorr.anc.gz scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: stderrgfx.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_3.gz No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting scan: 0mct_05987932_atmos.day.gz scan: 0mct_05987932_ocean.day.gz scan: 1029_flux_corr.anc.gz scan: boinc_lockfile scan: boinc_task_state.xml scan: boinc_ufs_cpdnout1.zip scan: boinc_ufs_cpdnout2.zip scan: boinc_ufs_cpdnout3.zip scan: boinc_ufs_cpdnout4.zip scan: cpdnout1.zip scan: cpdnout2.zip scan: cpdnout3.zip scan: cpdnout4.zip scan: cpdnout5.zip scan: cpdnout6.zip scan: cpdnout7.zip scan: cpdnout8.zip scan: DMSNH3SO2A1B_19182082N.gz scan: DMSNH3SO2A1B_19182082N.mod.gz scan: ghg_a1b.gz scan: graphics_app scan: hadcm3igeo_w30e_2000_80_07192722.zip scan: hadcm3trans_6.04_windows_intelx86.exe scan: hadcm3trans_data_6.04_windows_intelx86.zip scan: hadcm3trans_se_6.04_windows_intelx86.zip scan: hadcm3trans_um_6.04_windows_intelx86.zip scan: heblhdck_0208_nickfluxcorr.anc.gz scan: init_data.xml scan: NAT_VOLC.gz scan: ozone_hadcm3_1919_2082.gz scan: ozone_hadcm3_1919_2082.mod.gz scan: solar_v01.gz scan: spec3a_lw_3_asol2c_hadcm3.gz scan: spec3a_sw_3_asol2b_hadcm3.gz scan: stderr.txt scan: stderrgfx.txt scan: SULPC_OXIDANTS_19_A2_1990.gz scan: SULPC_OXIDANTS_19_A2_1990.mod.gz scan: volc_geo2_3.gz </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 16:30:05	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	1,218,240	2,368,438	1.9441
25 Jul 2011 15:54:10	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	1,192,320	2,319,448	1.9453
25 Jul 2011 15:29:59	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	1,166,400	2,279,405	1.9542
25 Jul 2011 14:49:18	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	1,140,480	2,240,802	1.9648
25 Jul 2011 13:31:12	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	1,114,560	2,198,993	1.9730
25 Jul 2011 13:31:12	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	1,088,640	2,148,553	1.9736
25 Jul 2011 13:31:12	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	1,062,720	2,097,392	1.9736
25 Jul 2011 13:31:12	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	1,036,800	2,045,071	1.9725
25 Jul 2011 13:31:12	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	1,010,880	1,991,677	1.9702
25 Jul 2011 13:31:12	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	984,960	1,939,234	1.9688
11 Jul 2011 03:16:12	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	959,040	1,888,743	1.9694
10 Jul 2011 11:07:45	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	933,120	1,836,410	1.9680
09 Jul 2011 18:35:29	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	907,200	1,783,973	1.9665
09 Jul 2011 01:46:51	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	881,280	1,732,131	1.9655
08 Jul 2011 08:53:54	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	855,360	1,679,396	1.9634
07 Jul 2011 17:45:58	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	829,440	1,627,017	1.9616
07 Jul 2011 15:38:08	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	803,520	1,575,107	1.9603
06 Jul 2011 01:48:18	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	777,600	1,522,835	1.9584
05 Jul 2011 07:38:26	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	751,680	1,469,437	1.9549
04 Jul 2011 15:28:08	1116868	12711959	hadcm3igeo_w30e_2000_80_07192722_0	725,760	1,417,931	1.9537