climateprediction.net home page
Task 22306805

Task 22306805

Name oifs_43r3_ps_0164_2021050100_123_990_12206656_0
Workunit 12206656
Created 9 Feb 2023, 15:05:38 UTC
Sent 9 Feb 2023, 15:11:48 UTC
Report deadline 10 Apr 2023, 15:11:48 UTC
Received 11 Feb 2023, 10:19:05 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1508075
Run time 8 hours 1 min 52 sec
CPU time 8 hours 0 min 10 sec
Validate state Invalid
Credit 11,612.16
Device peak FLOPS 3.65 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.09
x86_64-pc-linux-gnu
Peak working set size 4,419.00 MB
Peak swap size 4,978.08 MB
Peak disk usage 1,216.35 MB
Stderr
<core_client_version>7.23.0</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
tion.net/oifs_43r3_ps_12206656/ICMGGhpi1+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_78.zip
Uploading the intermediate file: upload_file_78.zip
  08:47:08 STEP 1897 H=1897:00 +CPU=  9.827
  08:47:18 STEP 1898 H=1898:00 +CPU= 10.094
  08:47:34 STEP 1899 H=1899:00 +CPU= 15.754
  08:47:44 STEP 1900 H=1900:00 +CPU= 10.151
  08:47:54 STEP 1901 H=1901:00 +CPU=  9.839
  08:48:10 STEP 1902 H=1902:00 +CPU= 16.103
  08:48:20 STEP 1903 H=1903:00 +CPU=  9.862
  08:48:30 STEP 1904 H=1904:00 +CPU= 10.125
  08:48:46 STEP 1905 H=1905:00 +CPU= 15.881
  08:48:56 STEP 1906 H=1906:00 +CPU= 10.166
  08:49:06 STEP 1907 H=1907:00 +CPU=  9.915
  08:49:24 STEP 1908 H=1908:00 +CPU= 17.719
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+001908
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+001908
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+001908
  08:49:34 STEP 1909 H=1909:00 +CPU=  9.850
  08:49:44 STEP 1910 H=1910:00 +CPU= 10.177
  08:50:00 STEP 1911 H=1911:00 +CPU= 15.832
  08:50:10 STEP 1912 H=1912:00 +CPU= 10.198
  08:50:20 STEP 1913 H=1913:00 +CPU=  9.891
  08:50:36 STEP 1914 H=1914:00 +CPU= 16.181
  08:50:46 STEP 1915 H=1915:00 +CPU=  9.887
  08:50:56 STEP 1916 H=1916:00 +CPU= 10.114
  08:51:12 STEP 1917 H=1917:00 +CPU= 15.886
  08:51:22 STEP 1918 H=1918:00 +CPU= 10.095
  08:51:32 STEP 1919 H=1919:00 +CPU= 10.198
  08:51:51 STEP 1920 H=1920:00 +CPU= 19.093
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+001920
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+001920
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_79.zip
Uploading the intermediate file: upload_file_79.zip
  08:52:01 STEP 1921 H=1921:00 +CPU=  9.896
  08:52:11 STEP 1922 H=1922:00 +CPU= 10.097
  08:52:27 STEP 1923 H=1923:00 +CPU= 15.848
  08:52:37 STEP 1924 H=1924:00 +CPU= 10.057
  08:52:47 STEP 1925 H=1925:00 +CPU=  9.909
  08:53:03 STEP 1926 H=1926:00 +CPU= 16.133
  08:53:13 STEP 1927 H=1927:00 +CPU=  9.920
  08:53:23 STEP 1928 H=1928:00 +CPU= 10.190
  08:53:39 STEP 1929 H=1929:00 +CPU= 15.819
  08:53:49 STEP 1930 H=1930:00 +CPU= 10.148
  08:53:59 STEP 1931 H=1931:00 +CPU=  9.883
  08:54:17 STEP 1932 H=1932:00 +CPU= 17.710
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+001932
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+001932
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+001932
  08:54:27 STEP 1933 H=1933:00 +CPU=  9.917
  08:54:37 STEP 1934 H=1934:00 +CPU= 10.145
  08:54:53 STEP 1935 H=1935:00 +CPU= 15.746
  08:55:03 STEP 1936 H=1936:00 +CPU= 10.143
  08:55:13 STEP 1937 H=1937:00 +CPU=  9.939
  08:55:29 STEP 1938 H=1938:00 +CPU= 16.103
  08:55:39 STEP 1939 H=1939:00 +CPU=  9.832
  08:55:49 STEP 1940 H=1940:00 +CPU= 10.119
  08:56:05 STEP 1941 H=1941:00 +CPU= 15.800
  08:56:15 STEP 1942 H=1942:00 +CPU= 10.128
  08:56:25 STEP 1943 H=1943:00 +CPU= 10.164
  08:56:44 STEP 1944 H=1944:00 +CPU= 19.166
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+001944
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+001944
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_80.zip
Uploading the intermediate file: upload_file_80.zip
  08:56:54 STEP 1945 H=1945:00 +CPU=  9.861
  08:57:04 STEP 1946 H=1946:00 +CPU= 10.124
  08:57:20 STEP 1947 H=1947:00 +CPU= 15.801
  08:57:30 STEP 1948 H=1948:00 +CPU= 10.142
  08:57:40 STEP 1949 H=1949:00 +CPU=  9.865
  08:57:56 STEP 1950 H=1950:00 +CPU= 16.222
  08:58:06 STEP 1951 H=1951:00 +CPU=  9.894
  08:58:16 STEP 1952 H=1952:00 +CPU= 10.211
  08:58:32 STEP 1953 H=1953:00 +CPU= 15.903
  08:58:43 STEP 1954 H=1954:00 +CPU= 10.175
  08:58:52 STEP 1955 H=1955:00 +CPU=  9.934
  08:59:10 STEP 1956 H=1956:00 +CPU= 17.803
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+001956
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+001956
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+001956
  08:59:20 STEP 1957 H=1957:00 +CPU=  9.871
  08:59:30 STEP 1958 H=1958:00 +CPU= 10.135
  08:59:46 STEP 1959 H=1959:00 +CPU= 15.819
  08:59:56 STEP 1960 H=1960:00 +CPU= 10.137
  09:00:06 STEP 1961 H=1961:00 +CPU=  9.922
  09:00:22 STEP 1962 H=1962:00 +CPU= 16.112
  09:00:32 STEP 1963 H=1963:00 +CPU=  9.853
  09:00:42 STEP 1964 H=1964:00 +CPU= 10.145
  09:00:58 STEP 1965 H=1965:00 +CPU= 15.733
  09:01:08 STEP 1966 H=1966:00 +CPU= 10.194
  09:01:18 STEP 1967 H=1967:00 +CPU= 10.165
  09:01:38 STEP 1968 H=1968:00 +CPU= 19.115
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+001968
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+001968
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_81.zip
  09:01:47 STEP 1969 H=1969:00 +CPU=  9.836
Uploading the intermediate file: upload_file_81.zip
  09:01:58 STEP 1970 H=1970:00 +CPU= 10.162
  09:02:13 STEP 1971 H=1971:00 +CPU= 15.794
  09:02:23 STEP 1972 H=1972:00 +CPU= 10.068
  09:02:33 STEP 1973 H=1973:00 +CPU=  9.822
  09:02:49 STEP 1974 H=1974:00 +CPU= 16.204
  09:02:59 STEP 1975 H=1975:00 +CPU=  9.954
  09:03:10 STEP 1976 H=1976:00 +CPU= 10.214
  09:03:25 STEP 1977 H=1977:00 +CPU= 15.764
  09:03:36 STEP 1978 H=1978:00 +CPU= 10.166
  09:03:45 STEP 1979 H=1979:00 +CPU=  9.915
  09:04:03 STEP 1980 H=1980:00 +CPU= 17.708
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+001980
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+001980
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+001980
  09:04:13 STEP 1981 H=1981:00 +CPU=  9.933
  09:04:23 STEP 1982 H=1982:00 +CPU= 10.123
  09:04:39 STEP 1983 H=1983:00 +CPU= 15.813
  09:04:49 STEP 1984 H=1984:00 +CPU= 10.149
  09:04:59 STEP 1985 H=1985:00 +CPU=  9.879
  09:05:15 STEP 1986 H=1986:00 +CPU= 16.236
  09:05:25 STEP 1987 H=1987:00 +CPU=  9.905
  09:05:35 STEP 1988 H=1988:00 +CPU= 10.156
  09:05:51 STEP 1989 H=1989:00 +CPU= 15.861
  09:06:01 STEP 1990 H=1990:00 +CPU= 10.126
  09:06:12 STEP 1991 H=1991:00 +CPU= 10.212
  09:06:31 STEP 1992 H=1992:00 +CPU= 19.161
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+001992
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+001992
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_82.zip
Uploading the intermediate file: upload_file_82.zip
  09:06:41 STEP 1993 H=1993:00 +CPU=  9.869
  09:06:51 STEP 1994 H=1994:00 +CPU= 10.158
  09:07:07 STEP 1995 H=1995:00 +CPU= 15.882
  09:07:17 STEP 1996 H=1996:00 +CPU= 10.175
  09:07:27 STEP 1997 H=1997:00 +CPU=  9.904
  09:07:43 STEP 1998 H=1998:00 +CPU= 16.229
  09:07:53 STEP 1999 H=1999:00 +CPU=  9.868
  09:08:03 STEP 2000 H=2000:00 +CPU= 10.215
  09:08:19 STEP 2001 H=2001:00 +CPU= 15.828
  09:08:29 STEP 2002 H=2002:00 +CPU= 10.187
  09:08:39 STEP 2003 H=2003:00 +CPU=  9.921
  09:08:57 STEP 2004 H=2004:00 +CPU= 17.830
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002004
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002004
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002004
  09:09:07 STEP 2005 H=2005:00 +CPU=  9.904
  09:09:17 STEP 2006 H=2006:00 +CPU= 10.175
  09:09:33 STEP 2007 H=2007:00 +CPU= 15.836
  09:09:43 STEP 2008 H=2008:00 +CPU= 10.200
  09:09:53 STEP 2009 H=2009:00 +CPU=  9.865
  09:10:09 STEP 2010 H=2010:00 +CPU= 16.108
  09:10:19 STEP 2011 H=2011:00 +CPU=  9.893
  09:10:29 STEP 2012 H=2012:00 +CPU= 10.133
  09:10:45 STEP 2013 H=2013:00 +CPU= 15.827
  09:10:55 STEP 2014 H=2014:00 +CPU= 10.164
  09:11:05 STEP 2015 H=2015:00 +CPU= 10.183
  09:11:24 STEP 2016 H=2016:00 +CPU= 19.221
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002016
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002016
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_83.zip
Uploading the intermediate file: upload_file_83.zip
  09:11:34 STEP 2017 H=2017:00 +CPU=  9.836
  09:11:44 STEP 2018 H=2018:00 +CPU= 10.100
  09:12:00 STEP 2019 H=2019:00 +CPU= 15.704
  09:12:10 STEP 2020 H=2020:00 +CPU= 10.180
  09:12:20 STEP 2021 H=2021:00 +CPU=  9.869
  09:12:36 STEP 2022 H=2022:00 +CPU= 16.209
  09:12:46 STEP 2023 H=2023:00 +CPU=  9.944
  09:12:57 STEP 2024 H=2024:00 +CPU= 10.186
  09:13:12 STEP 2025 H=2025:00 +CPU= 15.841
  09:13:23 STEP 2026 H=2026:00 +CPU= 10.168
  09:13:32 STEP 2027 H=2027:00 +CPU=  9.819
  09:13:50 STEP 2028 H=2028:00 +CPU= 17.696
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002028
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002028
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002028
  09:14:00 STEP 2029 H=2029:00 +CPU=  9.839
  09:14:10 STEP 2030 H=2030:00 +CPU= 10.171
  09:14:26 STEP 2031 H=2031:00 +CPU= 15.883
  09:14:36 STEP 2032 H=2032:00 +CPU= 10.183
  09:14:46 STEP 2033 H=2033:00 +CPU=  9.894
  09:15:02 STEP 2034 H=2034:00 +CPU= 16.163
  09:15:12 STEP 2035 H=2035:00 +CPU=  9.889
  09:15:22 STEP 2036 H=2036:00 +CPU= 10.187
  09:15:38 STEP 2037 H=2037:00 +CPU= 15.767
  09:15:48 STEP 2038 H=2038:00 +CPU= 10.164
  09:15:58 STEP 2039 H=2039:00 +CPU= 10.180
  09:16:18 STEP 2040 H=2040:00 +CPU= 19.106
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002040
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002040
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_84.zip
Uploading the intermediate file: upload_file_84.zip
  09:16:27 STEP 2041 H=2041:00 +CPU=  9.916
  09:16:38 STEP 2042 H=2042:00 +CPU= 10.222
  09:16:54 STEP 2043 H=2043:00 +CPU= 15.839
  09:17:04 STEP 2044 H=2044:00 +CPU= 10.100
  09:17:14 STEP 2045 H=2045:00 +CPU=  9.892
  09:17:30 STEP 2046 H=2046:00 +CPU= 16.132
  09:17:40 STEP 2047 H=2047:00 +CPU=  9.946
  09:17:50 STEP 2048 H=2048:00 +CPU= 10.205
  09:18:06 STEP 2049 H=2049:00 +CPU= 15.918
  09:18:16 STEP 2050 H=2050:00 +CPU= 10.177
  09:18:26 STEP 2051 H=2051:00 +CPU=  9.971
  09:18:44 STEP 2052 H=2052:00 +CPU= 17.852
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002052
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002052
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002052
  09:18:54 STEP 2053 H=2053:00 +CPU=  9.967
  09:19:04 STEP 2054 H=2054:00 +CPU= 10.242
  09:19:20 STEP 2055 H=2055:00 +CPU= 15.832
  09:19:30 STEP 2056 H=2056:00 +CPU= 10.194
  09:19:40 STEP 2057 H=2057:00 +CPU=  9.897
  09:19:56 STEP 2058 H=2058:00 +CPU= 16.141
  09:20:06 STEP 2059 H=2059:00 +CPU=  9.897
  09:20:16 STEP 2060 H=2060:00 +CPU= 10.071
  09:20:32 STEP 2061 H=2061:00 +CPU= 15.727
  09:20:42 STEP 2062 H=2062:00 +CPU= 10.162
  09:20:52 STEP 2063 H=2063:00 +CPU= 10.111
  09:21:11 STEP 2064 H=2064:00 +CPU= 19.151
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002064
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002064
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_85.zip
  09:21:21 STEP 2065 H=2065:00 +CPU=  9.848
Uploading the intermediate file: upload_file_85.zip
  09:21:31 STEP 2066 H=2066:00 +CPU= 10.162
Uploading trickle at timestep: 7430400
  09:21:47 STEP 2067 H=2067:00 +CPU= 15.846
  09:21:57 STEP 2068 H=2068:00 +CPU= 10.153
  09:22:07 STEP 2069 H=2069:00 +CPU=  9.931
  09:22:23 STEP 2070 H=2070:00 +CPU= 16.218
  09:22:33 STEP 2071 H=2071:00 +CPU=  9.883
  09:22:43 STEP 2072 H=2072:00 +CPU= 10.169
  09:22:59 STEP 2073 H=2073:00 +CPU= 15.742
  09:23:09 STEP 2074 H=2074:00 +CPU= 10.247
  09:23:19 STEP 2075 H=2075:00 +CPU=  9.906
  09:23:37 STEP 2076 H=2076:00 +CPU= 17.870
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002076
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002076
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002076
  09:23:47 STEP 2077 H=2077:00 +CPU=  9.947
  09:23:57 STEP 2078 H=2078:00 +CPU= 10.151
  09:24:13 STEP 2079 H=2079:00 +CPU= 15.868
  09:24:23 STEP 2080 H=2080:00 +CPU= 10.250
  09:24:33 STEP 2081 H=2081:00 +CPU= 10.004
  09:24:50 STEP 2082 H=2082:00 +CPU= 16.294
  09:25:00 STEP 2083 H=2083:00 +CPU=  9.855
  09:25:10 STEP 2084 H=2084:00 +CPU= 10.092
  09:25:26 STEP 2085 H=2085:00 +CPU= 15.780
  09:25:36 STEP 2086 H=2086:00 +CPU= 10.130
  09:25:46 STEP 2087 H=2087:00 +CPU= 10.142
  09:26:05 STEP 2088 H=2088:00 +CPU= 19.199
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002088
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002088
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_86.zip
Uploading the intermediate file: upload_file_86.zip
  09:26:15 STEP 2089 H=2089:00 +CPU=  9.943
  09:26:25 STEP 2090 H=2090:00 +CPU= 10.210
  09:26:41 STEP 2091 H=2091:00 +CPU= 15.799
  09:26:51 STEP 2092 H=2092:00 +CPU= 10.192
  09:27:01 STEP 2093 H=2093:00 +CPU=  9.947
  09:27:17 STEP 2094 H=2094:00 +CPU= 16.164
  09:27:27 STEP 2095 H=2095:00 +CPU=  9.919
  09:27:37 STEP 2096 H=2096:00 +CPU= 10.222
  09:27:53 STEP 2097 H=2097:00 +CPU= 15.912
  09:28:04 STEP 2098 H=2098:00 +CPU= 10.163
  09:28:13 STEP 2099 H=2099:00 +CPU=  9.904
  09:28:31 STEP 2100 H=2100:00 +CPU= 17.767
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002100
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002100
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002100
  09:28:41 STEP 2101 H=2101:00 +CPU=  9.952
  09:28:51 STEP 2102 H=2102:00 +CPU= 10.178
  09:29:07 STEP 2103 H=2103:00 +CPU= 15.724
  09:29:17 STEP 2104 H=2104:00 +CPU= 10.168
  09:29:27 STEP 2105 H=2105:00 +CPU=  9.852
  09:29:43 STEP 2106 H=2106:00 +CPU= 16.162
  09:29:53 STEP 2107 H=2107:00 +CPU=  9.939
  09:30:03 STEP 2108 H=2108:00 +CPU= 10.159
  09:30:19 STEP 2109 H=2109:00 +CPU= 15.809
  09:30:29 STEP 2110 H=2110:00 +CPU= 10.185
  09:30:39 STEP 2111 H=2111:00 +CPU= 10.126
  09:30:59 STEP 2112 H=2112:00 +CPU= 19.142
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002112
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002112
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_87.zip
Uploading the intermediate file: upload_file_87.zip
  09:31:08 STEP 2113 H=2113:00 +CPU=  9.812
  09:31:19 STEP 2114 H=2114:00 +CPU= 10.144
  09:31:34 STEP 2115 H=2115:00 +CPU= 15.827
  09:31:45 STEP 2116 H=2116:00 +CPU= 10.178
  09:31:55 STEP 2117 H=2117:00 +CPU=  9.900
  09:32:11 STEP 2118 H=2118:00 +CPU= 16.163
  09:32:21 STEP 2119 H=2119:00 +CPU=  9.898
  09:32:31 STEP 2120 H=2120:00 +CPU= 10.228
  09:32:47 STEP 2121 H=2121:00 +CPU= 15.822
  09:32:57 STEP 2122 H=2122:00 +CPU= 10.208
  09:33:07 STEP 2123 H=2123:00 +CPU=  9.993
  09:33:25 STEP 2124 H=2124:00 +CPU= 17.736
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002124
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002124
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002124
  09:33:34 STEP 2125 H=2125:00 +CPU=  9.889
  09:33:45 STEP 2126 H=2126:00 +CPU= 10.172
  09:34:01 STEP 2127 H=2127:00 +CPU= 15.846
  09:34:11 STEP 2128 H=2128:00 +CPU= 10.103
  09:34:21 STEP 2129 H=2129:00 +CPU=  9.908
  09:34:37 STEP 2130 H=2130:00 +CPU= 16.171
  09:34:47 STEP 2131 H=2131:00 +CPU=  9.949
  09:34:57 STEP 2132 H=2132:00 +CPU= 10.182
  09:35:13 STEP 2133 H=2133:00 +CPU= 15.837
  09:35:23 STEP 2134 H=2134:00 +CPU= 10.112
  09:35:33 STEP 2135 H=2135:00 +CPU= 10.129
  09:35:52 STEP 2136 H=2136:00 +CPU= 19.195
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002136
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002136
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_88.zip
Uploading the intermediate file: upload_file_88.zip
  09:36:02 STEP 2137 H=2137:00 +CPU=  9.853
  09:36:12 STEP 2138 H=2138:00 +CPU= 10.103
  09:36:28 STEP 2139 H=2139:00 +CPU= 15.941
  09:36:38 STEP 2140 H=2140:00 +CPU= 10.238
  09:36:48 STEP 2141 H=2141:00 +CPU=  9.934
  09:37:04 STEP 2142 H=2142:00 +CPU= 16.097
  09:37:14 STEP 2143 H=2143:00 +CPU=  9.906
  09:37:24 STEP 2144 H=2144:00 +CPU= 10.226
  09:37:40 STEP 2145 H=2145:00 +CPU= 15.878
  09:37:51 STEP 2146 H=2146:00 +CPU= 10.226
  09:38:00 STEP 2147 H=2147:00 +CPU=  9.887
  09:38:18 STEP 2148 H=2148:00 +CPU= 17.856
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002148
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002148
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002148
  09:38:28 STEP 2149 H=2149:00 +CPU=  9.859
  09:38:38 STEP 2150 H=2150:00 +CPU= 10.222
  09:38:54 STEP 2151 H=2151:00 +CPU= 15.834
  09:39:04 STEP 2152 H=2152:00 +CPU= 10.191
  09:39:14 STEP 2153 H=2153:00 +CPU=  9.836
  09:39:31 STEP 2154 H=2154:00 +CPU= 16.516
  09:39:41 STEP 2155 H=2155:00 +CPU= 10.264
  09:39:52 STEP 2156 H=2156:00 +CPU= 10.442
  09:40:08 STEP 2157 H=2157:00 +CPU= 16.180
  09:40:18 STEP 2158 H=2158:00 +CPU= 10.242
  09:40:28 STEP 2159 H=2159:00 +CPU= 10.254
  09:40:48 STEP 2160 H=2160:00 +CPU= 19.504
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002160
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002160
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_89.zip
Uploading the intermediate file: upload_file_89.zip
  09:40:58 STEP 2161 H=2161:00 +CPU= 10.055
  09:41:08 STEP 2162 H=2162:00 +CPU= 10.301
  09:41:24 STEP 2163 H=2163:00 +CPU= 15.908
  09:41:34 STEP 2164 H=2164:00 +CPU= 10.206
  09:41:44 STEP 2165 H=2165:00 +CPU=  9.903
  09:42:00 STEP 2166 H=2166:00 +CPU= 16.233
  09:42:10 STEP 2167 H=2167:00 +CPU=  9.963
  09:42:21 STEP 2168 H=2168:00 +CPU= 10.445
  09:42:37 STEP 2169 H=2169:00 +CPU= 16.335
  09:42:48 STEP 2170 H=2170:00 +CPU= 10.333
  09:42:58 STEP 2171 H=2171:00 +CPU= 10.070
  09:43:16 STEP 2172 H=2172:00 +CPU= 18.004
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002172
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002172
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002172
  09:43:26 STEP 2173 H=2173:00 +CPU= 10.032
  09:43:36 STEP 2174 H=2174:00 +CPU= 10.332
  09:43:52 STEP 2175 H=2175:00 +CPU= 15.946
  09:44:02 STEP 2176 H=2176:00 +CPU= 10.246
  09:44:12 STEP 2177 H=2177:00 +CPU=  9.990
  09:44:29 STEP 2178 H=2178:00 +CPU= 16.327
  09:44:39 STEP 2179 H=2179:00 +CPU= 10.046
  09:44:49 STEP 2180 H=2180:00 +CPU= 10.236
  09:45:05 STEP 2181 H=2181:00 +CPU= 16.266
  09:45:15 STEP 2182 H=2182:00 +CPU= 10.231
  09:45:26 STEP 2183 H=2183:00 +CPU= 10.344
  09:45:45 STEP 2184 H=2184:00 +CPU= 19.460
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002184
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002184
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_90.zip
  09:45:55 STEP 2185 H=2185:00 +CPU=  9.977
Uploading the intermediate file: upload_file_90.zip
  09:46:05 STEP 2186 H=2186:00 +CPU= 10.234
  09:46:21 STEP 2187 H=2187:00 +CPU= 15.959
  09:46:32 STEP 2188 H=2188:00 +CPU= 10.300
  09:46:42 STEP 2189 H=2189:00 +CPU= 10.027
  09:46:58 STEP 2190 H=2190:00 +CPU= 16.440
  09:47:08 STEP 2191 H=2191:00 +CPU=  9.958
  09:47:18 STEP 2192 H=2192:00 +CPU= 10.235
  09:47:34 STEP 2193 H=2193:00 +CPU= 15.852
  09:47:44 STEP 2194 H=2194:00 +CPU= 10.196
  09:47:54 STEP 2195 H=2195:00 +CPU=  9.957
  09:48:12 STEP 2196 H=2196:00 +CPU= 17.825
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002196
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002196
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002196
  09:48:22 STEP 2197 H=2197:00 +CPU=  9.985
  09:48:32 STEP 2198 H=2198:00 +CPU= 10.276
  09:48:48 STEP 2199 H=2199:00 +CPU= 15.876
  09:48:59 STEP 2200 H=2200:00 +CPU= 10.184
  09:49:08 STEP 2201 H=2201:00 +CPU=  9.935
  09:49:25 STEP 2202 H=2202:00 +CPU= 16.133
  09:49:35 STEP 2203 H=2203:00 +CPU=  9.911
  09:49:45 STEP 2204 H=2204:00 +CPU= 10.233
  09:50:01 STEP 2205 H=2205:00 +CPU= 15.921
  09:50:11 STEP 2206 H=2206:00 +CPU= 10.192
  09:50:21 STEP 2207 H=2207:00 +CPU= 10.146
  09:50:40 STEP 2208 H=2208:00 +CPU= 19.160
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002208
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002208
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_91.zip
Uploading the intermediate file: upload_file_91.zip
  09:50:50 STEP 2209 H=2209:00 +CPU=  9.884
  09:51:00 STEP 2210 H=2210:00 +CPU= 10.179
  09:51:16 STEP 2211 H=2211:00 +CPU= 15.947
  09:51:27 STEP 2212 H=2212:00 +CPU= 10.383
  09:51:37 STEP 2213 H=2213:00 +CPU= 10.092
  09:51:53 STEP 2214 H=2214:00 +CPU= 16.554
  09:52:03 STEP 2215 H=2215:00 +CPU=  9.974
  09:52:14 STEP 2216 H=2216:00 +CPU= 10.322
  09:52:30 STEP 2217 H=2217:00 +CPU= 16.201
  09:52:40 STEP 2218 H=2218:00 +CPU= 10.330
  09:52:50 STEP 2219 H=2219:00 +CPU= 10.248
  09:53:09 STEP 2220 H=2220:00 +CPU= 18.431
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002220
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002220
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002220
  09:53:19 STEP 2221 H=2221:00 +CPU= 10.189
  09:53:30 STEP 2222 H=2222:00 +CPU= 10.409
  09:53:46 STEP 2223 H=2223:00 +CPU= 16.179
  09:53:56 STEP 2224 H=2224:00 +CPU= 10.271
  09:54:06 STEP 2225 H=2225:00 +CPU= 10.023
  09:54:22 STEP 2226 H=2226:00 +CPU= 16.327
  09:54:32 STEP 2227 H=2227:00 +CPU=  9.968
  09:54:43 STEP 2228 H=2228:00 +CPU= 10.249
  09:54:59 STEP 2229 H=2229:00 +CPU= 16.105
  09:55:09 STEP 2230 H=2230:00 +CPU= 10.383
  09:55:19 STEP 2231 H=2231:00 +CPU= 10.288
  09:55:39 STEP 2232 H=2232:00 +CPU= 19.376
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002232
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002232
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_92.zip
Uploading the intermediate file: upload_file_92.zip
  09:55:49 STEP 2233 H=2233:00 +CPU= 10.005
  09:55:59 STEP 2234 H=2234:00 +CPU= 10.247
  09:56:15 STEP 2235 H=2235:00 +CPU= 15.988
  09:56:25 STEP 2236 H=2236:00 +CPU= 10.290
  09:56:35 STEP 2237 H=2237:00 +CPU= 10.041
  09:56:52 STEP 2238 H=2238:00 +CPU= 16.414
  09:57:02 STEP 2239 H=2239:00 +CPU= 10.054
  09:57:12 STEP 2240 H=2240:00 +CPU= 10.324
  09:57:28 STEP 2241 H=2241:00 +CPU= 16.158
  09:57:39 STEP 2242 H=2242:00 +CPU= 10.326
  09:57:49 STEP 2243 H=2243:00 +CPU= 10.035
  09:58:07 STEP 2244 H=2244:00 +CPU= 18.022
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002244
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002244
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002244
  09:58:17 STEP 2245 H=2245:00 +CPU= 10.089
  09:58:27 STEP 2246 H=2246:00 +CPU= 10.310
  09:58:43 STEP 2247 H=2247:00 +CPU= 16.061
  09:58:54 STEP 2248 H=2248:00 +CPU= 10.317
  09:59:04 STEP 2249 H=2249:00 +CPU= 10.056
  09:59:20 STEP 2250 H=2250:00 +CPU= 16.382
  09:59:30 STEP 2251 H=2251:00 +CPU=  9.977
  09:59:40 STEP 2252 H=2252:00 +CPU= 10.270
  09:59:56 STEP 2253 H=2253:00 +CPU= 16.045
  10:00:07 STEP 2254 H=2254:00 +CPU= 10.276
  10:00:17 STEP 2255 H=2255:00 +CPU= 10.353
  10:00:36 STEP 2256 H=2256:00 +CPU= 19.439
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002256
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002256
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_93.zip
Uploading the intermediate file: upload_file_93.zip
  10:00:46 STEP 2257 H=2257:00 +CPU=  9.962
  10:00:57 STEP 2258 H=2258:00 +CPU= 10.202
  10:01:13 STEP 2259 H=2259:00 +CPU= 16.053
  10:01:23 STEP 2260 H=2260:00 +CPU= 10.305
  10:01:33 STEP 2261 H=2261:00 +CPU= 10.003
  10:01:49 STEP 2262 H=2262:00 +CPU= 16.366
  10:01:59 STEP 2263 H=2263:00 +CPU=  9.956
  10:02:10 STEP 2264 H=2264:00 +CPU= 10.329
  10:02:26 STEP 2265 H=2265:00 +CPU= 16.092
  10:02:36 STEP 2266 H=2266:00 +CPU= 10.332
  10:02:46 STEP 2267 H=2267:00 +CPU= 10.102
  10:03:04 STEP 2268 H=2268:00 +CPU= 17.995
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002268
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002268
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002268
  10:03:14 STEP 2269 H=2269:00 +CPU= 10.124
  10:03:25 STEP 2270 H=2270:00 +CPU= 10.321
  10:03:41 STEP 2271 H=2271:00 +CPU= 16.029
  10:03:51 STEP 2272 H=2272:00 +CPU= 10.280
  10:04:01 STEP 2273 H=2273:00 +CPU= 10.025
  10:04:17 STEP 2274 H=2274:00 +CPU= 16.309
  10:04:27 STEP 2275 H=2275:00 +CPU=  9.999
  10:04:38 STEP 2276 H=2276:00 +CPU= 10.212
  10:04:54 STEP 2277 H=2277:00 +CPU= 16.026
  10:05:04 STEP 2278 H=2278:00 +CPU= 10.330
  10:05:14 STEP 2279 H=2279:00 +CPU= 10.325
  10:05:34 STEP 2280 H=2280:00 +CPU= 19.348
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002280
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002280
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_94.zip
Uploading the intermediate file: upload_file_94.zip
  10:05:44 STEP 2281 H=2281:00 +CPU= 10.076
  10:05:54 STEP 2282 H=2282:00 +CPU= 10.262
  10:06:10 STEP 2283 H=2283:00 +CPU= 16.021
  10:06:20 STEP 2284 H=2284:00 +CPU= 10.324
  10:06:30 STEP 2285 H=2285:00 +CPU= 10.038
  10:06:47 STEP 2286 H=2286:00 +CPU= 16.423
  10:06:57 STEP 2287 H=2287:00 +CPU= 10.000
  10:07:07 STEP 2288 H=2288:00 +CPU= 10.293
  10:07:23 STEP 2289 H=2289:00 +CPU= 15.967
  10:07:33 STEP 2290 H=2290:00 +CPU= 10.318
  10:07:43 STEP 2291 H=2291:00 +CPU= 10.043
  10:08:01 STEP 2292 H=2292:00 +CPU= 18.058
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002292
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002292
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002292
  10:08:12 STEP 2293 H=2293:00 +CPU= 10.100
  10:08:22 STEP 2294 H=2294:00 +CPU= 10.309
  10:08:38 STEP 2295 H=2295:00 +CPU= 16.049
  10:08:48 STEP 2296 H=2296:00 +CPU= 10.252
  10:08:58 STEP 2297 H=2297:00 +CPU= 10.053
  10:09:15 STEP 2298 H=2298:00 +CPU= 16.359
  10:09:25 STEP 2299 H=2299:00 +CPU= 10.045
  10:09:35 STEP 2300 H=2300:00 +CPU= 10.307
  10:09:51 STEP 2301 H=2301:00 +CPU= 16.049
  10:10:01 STEP 2302 H=2302:00 +CPU= 10.235
  10:10:12 STEP 2303 H=2303:00 +CPU= 10.337
  10:10:31 STEP 2304 H=2304:00 +CPU= 19.274
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhpi1+002304
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhpi1+002304
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhpi1+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMGGhpi1+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMSHhpi1+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206656/ICMUAhpi1+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0164_2021050100_123_990_12206656_0_r1268607713_95.zip
Uploading the intermediate file: upload_file_95.zip
  10:10:41 STEP 2305 H=2305:00 +CPU=  9.895
  10:10:51 STEP 2306 H=2306:00 +CPU= 10.282
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 4035MB (heap), 4420MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x815130 (old at (nil))
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point invalid operation [memaddr=0x1990af5], nsigs = 1
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4035 MB (maxheap), 4420 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.270] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.271] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.271] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.271] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.271] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.271] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.271] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.271] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-RADLSW       (1210) 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.271] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               RADIATION_SCHEME 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.271] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                radiation_interface:radiation 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101057:1676110257:14769.271] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 radiation_cloud_optics:cloud_optics 
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/5/oifs_43r3_model.exe' : sigcontextptr=0x7fff34e30850
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 21) with addr2line-cmd
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/5/oifs_43r3_model.exe' 0x839403 0x839a09 0x1dc91d0 0x1990af5 0x192d769 0xf5d7a5 0xe53815 0xe36114 0xc2abc9 0x134a8e2 0xbe93fe 0x4f8534 0x4b5c4c 0x49ef09 0x49e152 0x49e048 0x40708d 0x40220f 0x4021a2 0x1dca4d0 0x40206e
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839403] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839a09] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dc91d0] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1990af5] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x192d769] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0xf5d7a5] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0xe53815] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xe36114] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xc2abc9] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x134a8e2] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbe93fe] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x4f8534] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x4b5c4c] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x49ef09] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x49e152] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x49e048] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x40708d] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x40220f] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x4021a2] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x1dca4d0] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x40206e] : ??() at ??:0
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.271] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/5/oifs_43r3_model.exe' : sigcontextptr=0x7fff34e30760
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 22) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/5/oifs_43r3_model.exe' 0x839403 0x839a09 0x816de0 0x1dc91d0 0x1990af5 0x192d769 0xf5d7a5 0xe53815 0xe36114 0xc2abc9 0x134a8e2 0xbe93fe 0x4f8534 0x4b5c4c 0x49ef09 0x49e152 0x49e048 0x40708d 0x40220f 0x4021a2 0x1dca4d0 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839403] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839a09] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x816de0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dc91d0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1990af5] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x192d769] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0xf5d7a5] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xe53815] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xe36114] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xc2abc9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x134a8e2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbe93fe] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x4f8534] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x4b5c4c] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x49ef09] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x49e152] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x49e048] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x40708d] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40220f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x4021a2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x1dca4d0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x40206e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.286] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:swarm:1:1:4860:4860] [20230211:101058:1676110258:14770.286] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8cf0 for signal#8, nsigs = 1
forrtl: error (65): floating invalid
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE92AB  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000815D61  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC91D0  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001990AF5  radiation_cloud_o         400  radiation_cloud_optics.F90
oifs_43r3_model.e  000000000192D769  radiation_interfa         338  radiation_interface.F90
oifs_43r3_model.e  0000000000F5D7A5  radiation_scheme_         484  radiation_scheme.F90
oifs_43r3_model.e  0000000000E53815  radintg_                 1455  radintg.F90
oifs_43r3_model.e  0000000000E36114  raddrv_                   537  raddrv.F90
oifs_43r3_model.e  0000000000C2ABC9  gp_model_                 482  gp_model.F90
oifs_43r3_model.e  000000000134A8E2  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BE93FE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  00000000004F8534  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004B5C4C  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  000000000049EF09  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  000000000049E152  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049E048  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA4D0  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
Uploading trickle at timestep: 8294400
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -1.92518808265989E-02 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: Using seed      1057749417 to evolve YSPSDT_AR1(  1) in NSTEP=        2304
  STEPO: evolving YSPSDT_AR1(  1)
  MAXGPFV : MAX. VALUE =    1.11071791657739     
  MAXGPFV : MAX. VALUE =   0.702928622917406     
 NSTEP =  2304 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGhpi1+002304 MODE=a
 IO-STREAM CLOSED - ICMGGhpi1+002304
 MPI-TASK:   1 -    348380928 BYTES IN      1 RECORDS TRANSFERRED IN    0.0001 SECONDS******** Mbytes/s, TOTAL TIME=    0.0015( 6.7%)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210804231500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210804233000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210804234500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210805000000  !!!!!!!!!!!! 
 10:10:31 STEP 2304 H=2304:00 +CPU= 19.274
ISTEP=    2305 RSOLINC= 1360.9119 RI0= 1366.0000 ZI0= 1360.9119 RIP0= 1322.3785
ISTEP=    2305 ZI0= 1360.9119 ZSEASON= 0.971685593 REA=149597870000.00 RDEASO=151761820705.94
 NSTEP =  2305 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.145434711341E-01 1.792150403426E-01 5.886554914474E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.410156674493E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.86360955204255E-03 -1.83758561282373E+00  1.98826334025269E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.05368843416224E-03 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: evolving YSPSDT_AR1(  1)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210805001500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210805003000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210805004500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210805010000  !!!!!!!!!!!! 
 10:10:41 STEP 2305 H=2305:00 +CPU=  9.895
 NSTEP =  2306 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.198012218424E-01 1.784822145423E-01 5.823340849336E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.503447159555E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -9.02833338350058E-03 -1.81904082601515E+00  1.81055155589444E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -5.86992255146202E-03 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: evolving YSPSDT_AR1(  1)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210805011500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210805013000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210805014500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20210805020000  !!!!!!!!!!!! 
 10:10:51 STEP 2306 H=2306:00 +CPU= 10.282
  DATE= 2021    8    5
ISTEP=    2307: Reset RSOLINC= 1360.9119 RIP0= 1322.4105
 SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2:   6.299956272780167E-004
  9.481706056673667E-007  5.011088426183166E-007  9.312385019107499E-010
  1.993923492392666E-009  7.940283000000001E-011
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20210501
      138510  9.999999999999998E-005
SUECSO4: USE AVAILABLE FILES FOR DECADES:   2010 2020
 SUBROUTINE SUECSO4:, UNWEIGHTED AVERAGE:   2.954126739900534E-003
 NSTEP =  2307 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.190233379093E-01 1.784692685644E-01 5.887699341843E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.502425340923E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  1.64327784650500E-02 -1.96006002221036E+00  1.91368808362072E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  1.93404783180058E-02 -1.00000000000000E+00  1.00000000000000E+00
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 10:10:14 FULLPOS-B DYNFPOS   2304     0.410    0.410    0.410    245:19  -3333:** 0.11262231052220E-04       2GB       0MB
 10:10:14 FULLPOS-V DYNFPOS   2304     0.655    0.655    0.655    245:19  -3333:** 0.11262231052220E-04       2GB       0MB
 10:10:14 FULLPOS-T DYNFPOS   2304     0.229    0.229    0.229    245:20  -3333:** 0.11262231052220E-04       2GB       0MB
 10:10:15 0AAA00AAA STEPO     2304     0.227    0.227    0.227    245:20  -3333:** 0.11262231052220E-04       2GB       0MB
 10:10:31 0AAA00AAA STEPO     2305    16.242   16.242   16.247    245:36  -3333:** 0.11262231052220E-04       2GB       0MB
 10:10:41 0AAA00AAA STEPO     2306     9.895    9.895    9.900    245:46  -3333:** 0.11262231052220E-04       2GB       0MB
 10:10:51 0AAA00AAA STEPO     2307    10.283   10.283   10.288    245:56  -3333:** 0.11262231052220E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    2304",
 CTIME   = "00960000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 5*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =   98333.1287194678     ,
 GMASSI  =   98332.8725118276     
 /
------------------------------------------------
>>> Printing last 17 lines from file: waminfo

PARAM=10U/10V,                                                          
REPRES=GG,                                                              
ANALYSIS PERIOD =       0                                               
FORECAST PERIOD =    6307200                                            
WIND TIME STEP IN SECONDS =    3600                                     
ANALYSIS FROM 20210501000000 TO 20210501000000                          
ASSIMILATION NO                                                         
NEW FORECAST NO                                                         
STATUS STORM1= UNFINISHED                                               
STATUS STORM2= UNFINISHED                                               
STATUS STORM3= UNFINISHED                                               
DATE FOR OUTPUT OF BOTH RESTART FILES = 00000000000000                  
LAST DATE FOR SPECTRA FILE OUTPUT = 20210901000000                      
BEGIN DATE FOR USING SURFACE CURRENT = 20210501000000                   
PARALLEL RESTART = YES                                                  
NUMBER OF MPI TASKS USED =          1                                   
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/5/progress_file_12206656.xml

<running_values>
  <last_cpu_time>28790.590000</last_cpu_time>
  <upload_file_number>96</upload_file_number>
  <last_iter>2305</last_iter>
  <last_upload>8294400</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:50:35 1508075 22306805 oifs_43r3_ps_0164_2021050100_123_990_12206656_0 8,294,400 28,790 0.0035
09 May 2023 16:50:35 1508075 22306805 oifs_43r3_ps_0164_2021050100_123_990_12206656_0 7,430,400 25,833 0.0035
09 May 2023 16:50:35 1508075 22306805 oifs_43r3_ps_0164_2021050100_123_990_12206656_0 6,566,400 22,896 0.0035
09 May 2023 16:50:35 1508075 22306805 oifs_43r3_ps_0164_2021050100_123_990_12206656_0 5,702,400 19,969 0.0035
09 May 2023 16:50:34 1508075 22306805 oifs_43r3_ps_0164_2021050100_123_990_12206656_0 4,838,400 17,021 0.0035
09 May 2023 16:50:34 1508075 22306805 oifs_43r3_ps_0164_2021050100_123_990_12206656_0 3,456,000 12,244 0.0035
09 May 2023 16:50:30 1508075 22306805 oifs_43r3_ps_0164_2021050100_123_990_12206656_0 2,592,000 9,247 0.0036
09 May 2023 16:50:30 1508075 22306805 oifs_43r3_ps_0164_2021050100_123_990_12206656_0 1,728,000 6,260 0.0036
09 May 2023 16:50:30 1508075 22306805 oifs_43r3_ps_0164_2021050100_123_990_12206656_0 864,000 3,224 0.0037


©2024 climateprediction.net