climateprediction.net home page
Task 22263902

Task 22263902

Name oifs_43r3_ps_0366_1993050100_123_962_12179010_0
Workunit 12179010
Created 22 Dec 2022, 2:56:07 UTC
Sent 20 Jan 2023, 1:10:34 UTC
Report deadline 21 Mar 2023, 1:10:34 UTC
Received 20 Jan 2023, 17:33:10 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1532184
Run time 14 hours 25 min 31 sec
CPU time 14 hours 22 min 50 sec
Validate state Invalid
Credit 9,676.80
Device peak FLOPS 6.86 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,414.52 MB
Peak swap size 4,978.00 MB
Peak disk usage 1,218.70 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
2:42 STEP 1833 H=1833:00 +CPU= 32.854
  15:13:04 STEP 1834 H=1834:00 +CPU= 22.394
  15:13:26 STEP 1835 H=1835:00 +CPU= 21.870
  15:14:02 STEP 1836 H=1836:00 +CPU= 35.864
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001836
  15:14:23 STEP 1837 H=1837:00 +CPU= 21.461
  15:14:45 STEP 1838 H=1838:00 +CPU= 22.023
  15:15:18 STEP 1839 H=1839:00 +CPU= 32.905
  15:15:40 STEP 1840 H=1840:00 +CPU= 22.163
  15:16:02 STEP 1841 H=1841:00 +CPU= 21.368
  15:16:35 STEP 1842 H=1842:00 +CPU= 33.605
  15:16:57 STEP 1843 H=1843:00 +CPU= 21.668
  15:17:19 STEP 1844 H=1844:00 +CPU= 22.298
  15:17:52 STEP 1845 H=1845:00 +CPU= 32.930
  15:18:15 STEP 1846 H=1846:00 +CPU= 22.433
  15:18:37 STEP 1847 H=1847:00 +CPU= 22.672
  15:19:16 STEP 1848 H=1848:00 +CPU= 39.019
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0366_1993050100_123_962_12179010_0_r1505213178_76.zip
Uploading the intermediate file: upload_file_76.zip
  15:19:38 STEP 1849 H=1849:00 +CPU= 21.138
  15:19:59 STEP 1850 H=1850:00 +CPU= 21.594
  15:20:32 STEP 1851 H=1851:00 +CPU= 32.614
  15:20:54 STEP 1852 H=1852:00 +CPU= 22.104
  15:21:15 STEP 1853 H=1853:00 +CPU= 21.471
  15:21:49 STEP 1854 H=1854:00 +CPU= 33.376
  15:22:10 STEP 1855 H=1855:00 +CPU= 21.560
  15:22:32 STEP 1856 H=1856:00 +CPU= 22.078
  15:23:06 STEP 1857 H=1857:00 +CPU= 33.147
  15:23:28 STEP 1858 H=1858:00 +CPU= 22.345
  15:23:50 STEP 1859 H=1859:00 +CPU= 21.925
  15:24:26 STEP 1860 H=1860:00 +CPU= 35.952
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001860
  15:24:47 STEP 1861 H=1861:00 +CPU= 21.459
  15:25:09 STEP 1862 H=1862:00 +CPU= 21.937
  15:25:42 STEP 1863 H=1863:00 +CPU= 32.591
  15:26:04 STEP 1864 H=1864:00 +CPU= 22.298
  15:26:26 STEP 1865 H=1865:00 +CPU= 21.650
  15:26:59 STEP 1866 H=1866:00 +CPU= 33.378
  15:27:21 STEP 1867 H=1867:00 +CPU= 21.755
  15:27:43 STEP 1868 H=1868:00 +CPU= 22.215
  15:28:16 STEP 1869 H=1869:00 +CPU= 32.913
  15:28:39 STEP 1870 H=1870:00 +CPU= 22.303
  15:29:01 STEP 1871 H=1871:00 +CPU= 22.280
  15:29:40 STEP 1872 H=1872:00 +CPU= 38.846
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0366_1993050100_123_962_12179010_0_r1505213178_77.zip
Uploading the intermediate file: upload_file_77.zip
  15:30:01 STEP 1873 H=1873:00 +CPU= 21.529
  15:30:23 STEP 1874 H=1874:00 +CPU= 21.815
  15:30:56 STEP 1875 H=1875:00 +CPU= 33.246
  15:31:19 STEP 1876 H=1876:00 +CPU= 22.208
  15:31:41 STEP 1877 H=1877:00 +CPU= 21.929
  15:32:14 STEP 1878 H=1878:00 +CPU= 33.328
  15:32:36 STEP 1879 H=1879:00 +CPU= 21.754
  15:32:58 STEP 1880 H=1880:00 +CPU= 21.972
  15:33:30 STEP 1881 H=1881:00 +CPU= 32.797
  15:33:53 STEP 1882 H=1882:00 +CPU= 22.473
  15:34:15 STEP 1883 H=1883:00 +CPU= 22.025
  15:34:51 STEP 1884 H=1884:00 +CPU= 36.103
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001884
  15:35:13 STEP 1885 H=1885:00 +CPU= 21.732
  15:35:35 STEP 1886 H=1886:00 +CPU= 22.322
  15:36:08 STEP 1887 H=1887:00 +CPU= 32.697
  15:36:30 STEP 1888 H=1888:00 +CPU= 22.444
  15:36:52 STEP 1889 H=1889:00 +CPU= 21.294
  15:37:25 STEP 1890 H=1890:00 +CPU= 33.377
  15:37:47 STEP 1891 H=1891:00 +CPU= 21.632
  15:38:09 STEP 1892 H=1892:00 +CPU= 22.160
  15:38:42 STEP 1893 H=1893:00 +CPU= 32.873
  15:39:04 STEP 1894 H=1894:00 +CPU= 22.333
  15:39:26 STEP 1895 H=1895:00 +CPU= 22.141
  15:40:05 STEP 1896 H=1896:00 +CPU= 38.576
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0366_1993050100_123_962_12179010_0_r1505213178_78.zip
Uploading the intermediate file: upload_file_78.zip
  15:40:26 STEP 1897 H=1897:00 +CPU= 21.335
  15:40:48 STEP 1898 H=1898:00 +CPU= 21.849
  15:41:21 STEP 1899 H=1899:00 +CPU= 32.829
  15:41:43 STEP 1900 H=1900:00 +CPU= 22.039
  15:42:05 STEP 1901 H=1901:00 +CPU= 21.610
  15:42:38 STEP 1902 H=1902:00 +CPU= 33.492
  15:43:00 STEP 1903 H=1903:00 +CPU= 21.816
  15:43:22 STEP 1904 H=1904:00 +CPU= 22.134
  15:43:55 STEP 1905 H=1905:00 +CPU= 32.813
  15:44:17 STEP 1906 H=1906:00 +CPU= 22.271
  15:44:39 STEP 1907 H=1907:00 +CPU= 21.490
  15:45:15 STEP 1908 H=1908:00 +CPU= 36.037
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001908
  15:45:37 STEP 1909 H=1909:00 +CPU= 21.945
  15:46:00 STEP 1910 H=1910:00 +CPU= 22.552
  15:46:33 STEP 1911 H=1911:00 +CPU= 33.652
  15:46:56 STEP 1912 H=1912:00 +CPU= 22.267
  15:47:18 STEP 1913 H=1913:00 +CPU= 21.870
  15:47:52 STEP 1914 H=1914:00 +CPU= 33.911
  15:48:14 STEP 1915 H=1915:00 +CPU= 21.972
  15:48:36 STEP 1916 H=1916:00 +CPU= 22.419
  15:49:10 STEP 1917 H=1917:00 +CPU= 33.357
  15:49:33 STEP 1918 H=1918:00 +CPU= 23.285
  15:49:56 STEP 1919 H=1919:00 +CPU= 23.456
  15:50:37 STEP 1920 H=1920:00 +CPU= 40.207
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0366_1993050100_123_962_12179010_0_r1505213178_79.zip
Uploading the intermediate file: upload_file_79.zip
  15:50:59 STEP 1921 H=1921:00 +CPU= 22.028
Uploading trickle at timestep: 6912000
  15:51:21 STEP 1922 H=1922:00 +CPU= 22.511
  15:51:56 STEP 1923 H=1923:00 +CPU= 34.342
  15:52:19 STEP 1924 H=1924:00 +CPU= 22.978
  15:52:41 STEP 1925 H=1925:00 +CPU= 21.957
  15:53:15 STEP 1926 H=1926:00 +CPU= 34.686
  15:53:38 STEP 1927 H=1927:00 +CPU= 22.159
  15:54:00 STEP 1928 H=1928:00 +CPU= 22.367
  15:54:34 STEP 1929 H=1929:00 +CPU= 34.296
  15:54:57 STEP 1930 H=1930:00 +CPU= 22.806
  15:55:19 STEP 1931 H=1931:00 +CPU= 22.031
  15:55:57 STEP 1932 H=1932:00 +CPU= 37.347
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001932
  15:56:19 STEP 1933 H=1933:00 +CPU= 22.296
  15:56:41 STEP 1934 H=1934:00 +CPU= 22.442
  15:57:15 STEP 1935 H=1935:00 +CPU= 34.194
  15:57:39 STEP 1936 H=1936:00 +CPU= 23.250
  15:58:01 STEP 1937 H=1937:00 +CPU= 22.196
  15:58:36 STEP 1938 H=1938:00 +CPU= 34.618
  15:58:58 STEP 1939 H=1939:00 +CPU= 22.676
  15:59:21 STEP 1940 H=1940:00 +CPU= 23.028
  15:59:55 STEP 1941 H=1941:00 +CPU= 34.142
  16:00:19 STEP 1942 H=1942:00 +CPU= 23.338
  16:00:42 STEP 1943 H=1943:00 +CPU= 23.099
  16:01:22 STEP 1944 H=1944:00 +CPU= 40.369
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0366_1993050100_123_962_12179010_0_r1505213178_80.zip
Uploading the intermediate file: upload_file_80.zip
  16:01:45 STEP 1945 H=1945:00 +CPU= 22.180
  16:02:07 STEP 1946 H=1946:00 +CPU= 22.471
  16:02:41 STEP 1947 H=1947:00 +CPU= 33.814
  16:03:04 STEP 1948 H=1948:00 +CPU= 22.913
  16:03:26 STEP 1949 H=1949:00 +CPU= 22.040
  16:04:00 STEP 1950 H=1950:00 +CPU= 34.612
  16:04:23 STEP 1951 H=1951:00 +CPU= 22.330
  16:04:45 STEP 1952 H=1952:00 +CPU= 22.367
  16:05:19 STEP 1953 H=1953:00 +CPU= 33.963
  16:05:42 STEP 1954 H=1954:00 +CPU= 22.844
  16:06:04 STEP 1955 H=1955:00 +CPU= 22.024
  16:06:41 STEP 1956 H=1956:00 +CPU= 37.199
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001956
  16:07:04 STEP 1957 H=1957:00 +CPU= 22.367
  16:07:26 STEP 1958 H=1958:00 +CPU= 22.623
  16:08:01 STEP 1959 H=1959:00 +CPU= 34.112
  16:08:24 STEP 1960 H=1960:00 +CPU= 23.386
  16:08:46 STEP 1961 H=1961:00 +CPU= 22.275
  16:09:21 STEP 1962 H=1962:00 +CPU= 34.581
  16:09:44 STEP 1963 H=1963:00 +CPU= 22.753
  16:10:06 STEP 1964 H=1964:00 +CPU= 22.684
  16:10:40 STEP 1965 H=1965:00 +CPU= 33.856
  16:11:03 STEP 1966 H=1966:00 +CPU= 23.203
  16:11:26 STEP 1967 H=1967:00 +CPU= 22.944
  16:12:06 STEP 1968 H=1968:00 +CPU= 39.796
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0366_1993050100_123_962_12179010_0_r1505213178_81.zip
Uploading the intermediate file: upload_file_81.zip
  16:12:29 STEP 1969 H=1969:00 +CPU= 22.395
  16:12:51 STEP 1970 H=1970:00 +CPU= 22.538
  16:13:25 STEP 1971 H=1971:00 +CPU= 34.036
  16:13:48 STEP 1972 H=1972:00 +CPU= 23.167
  16:14:10 STEP 1973 H=1973:00 +CPU= 21.932
  16:14:45 STEP 1974 H=1974:00 +CPU= 34.890
  16:15:08 STEP 1975 H=1975:00 +CPU= 22.730
  16:15:31 STEP 1976 H=1976:00 +CPU= 22.876
  16:16:06 STEP 1977 H=1977:00 +CPU= 35.135
  16:16:30 STEP 1978 H=1978:00 +CPU= 23.911
  16:16:53 STEP 1979 H=1979:00 +CPU= 22.996
  16:17:31 STEP 1980 H=1980:00 +CPU= 38.464
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001980
  16:17:55 STEP 1981 H=1981:00 +CPU= 23.288
  16:18:14 STEP 1982 H=1982:00 +CPU= 19.758
  16:18:41 STEP 1983 H=1983:00 +CPU= 26.026
  16:18:57 STEP 1984 H=1984:00 +CPU= 16.946
  16:19:14 STEP 1985 H=1985:00 +CPU= 16.209
  16:19:40 STEP 1986 H=1986:00 +CPU= 26.361
  16:19:59 STEP 1987 H=1987:00 +CPU= 18.434
  16:20:16 STEP 1988 H=1988:00 +CPU= 17.241
  16:20:42 STEP 1989 H=1989:00 +CPU= 26.339
  16:21:00 STEP 1990 H=1990:00 +CPU= 17.801
  16:21:19 STEP 1991 H=1991:00 +CPU= 18.997
  16:21:58 STEP 1992 H=1992:00 +CPU= 38.946
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0366_1993050100_123_962_12179010_0_r1505213178_82.zip
Uploading the intermediate file: upload_file_82.zip
  16:22:20 STEP 1993 H=1993:00 +CPU= 21.599
  16:22:43 STEP 1994 H=1994:00 +CPU= 23.011
  16:23:17 STEP 1995 H=1995:00 +CPU= 34.252
  16:23:34 STEP 1996 H=1996:00 +CPU= 17.133
  16:23:50 STEP 1997 H=1997:00 +CPU= 16.450
  16:24:17 STEP 1998 H=1998:00 +CPU= 26.648
  16:24:35 STEP 1999 H=1999:00 +CPU= 17.903
  16:24:52 STEP 2000 H=2000:00 +CPU= 17.034
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4031MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4031 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.817] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.818] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.818] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.818] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162456:1674228296:51093.818] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-INPUT LOOP5  (1209) 
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162457:1674228297:51094.819] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:20202:20202] [20230120:162457:1674228297:51094.819] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001296C8B  radintg_                 1167  radintg.F90
oifs_43r3_model.e  0000000001270124  raddrv_                   537  raddrv.F90
oifs_43r3_model.e  0000000000BFD839  gp_model_                 482  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0366
(argv4) batchid: 962
(argv5) wuid: 12179010
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_96932c7733c61880ef0897837abedb3f to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0366_1993050100_123_962_12179010.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0366_1993050100_123_962_12179010.zip
ic_ancil_file: ic_ancil_12179010
ifsdata_file: ifsdata_12179010
climate_data_file: clim_data_12179010
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_13cb71b4dc22958601b9500812a29942 to: /var/lib/boinc-client/slots/3/ic_ancil_12179010.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12179010.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_88cb4faa6629e0c1a170ece815b6b324 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12179010.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12179010.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12179010.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12179010.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12179010.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12179010.xml
-- Model is restarting --
Adjusting last_iter, 2001, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12179010.xml
last_cpu_time: 51062
upload_file_number: 83
last_iter: 1991
last_upload: 7171200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0366_1993050100_123_962_12179010_0_r1505213178
The child process has been launched with process id: 1455
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171059:1674231059:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:12:01 STEP 1992 H=1992:00 +CPU= 45.014
  17:12:25 STEP 1993 H=1993:00 +CPU= 24.364
  17:12:47 STEP 1994 H=1994:00 +CPU= 21.767
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4032MB (heap), 4403MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4032 MB (maxheap), 4403 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
double free or corruption (top)
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-RADLSW       (1210) 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               RADIATION_SCHEME 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                radiation_interface:radiation 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 radiation_ifs_rrtm:gas_optics 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171253:1674231173:114.969] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  RRTM_GAS_OPTICAL_DEPTH 
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171255:1674231175:116.024] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:1455:1455] [20230120:171255:1674231175:116.025] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000193F703  rrtm_gas_optical_         182  rrtm_gas_optical_depth.F90
oifs_43r3_model.e  00000000018BA593  radiation_ifs_rrt         364  radiation_ifs_rrtm.F90
oifs_43r3_model.e  0000000001862293  radiation_interfa         311  radiation_interface.F90
oifs_43r3_model.e  00000000012D30E5  radiation_scheme_         484  radiation_scheme.F90
oifs_43r3_model.e  000000000129A415  radintg_                 1455  radintg.F90
oifs_43r3_model.e  0000000001270124  raddrv_                   537  raddrv.F90
oifs_43r3_model.e  0000000000BFD839  gp_model_                 482  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0366
(argv4) batchid: 962
(argv5) wuid: 12179010
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_96932c7733c61880ef0897837abedb3f to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0366_1993050100_123_962_12179010.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0366_1993050100_123_962_12179010.zip
ic_ancil_file: ic_ancil_12179010
ifsdata_file: ifsdata_12179010
climate_data_file: clim_data_12179010
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_13cb71b4dc22958601b9500812a29942 to: /var/lib/boinc-client/slots/3/ic_ancil_12179010.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12179010.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_88cb4faa6629e0c1a170ece815b6b324 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12179010.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12179010.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12179010.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12179010.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12179010.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12179010.xml
-- Model is restarting --
Adjusting last_iter, 1995, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12179010.xml
last_cpu_time: 51159
upload_file_number: 83
last_iter: 1991
last_upload: 7171200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0366_1993050100_123_962_12179010_0_r1505213178
The child process has been launched with process id: 1449
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1449:1449] [20230120:181219:1674234739:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:13:18 STEP 1992 H=1992:00 +CPU= 42.665
  18:13:40 STEP 1993 H=1993:00 +CPU= 21.905
  18:14:02 STEP 1994 H=1994:00 +CPU= 21.578
  18:14:33 STEP 1995 H=1995:00 +CPU= 31.833
  18:14:53 STEP 1996 H=1996:00 +CPU= 19.897
  18:15:14 STEP 1997 H=1997:00 +CPU= 20.862
  18:15:47 STEP 1998 H=1998:00 +CPU= 32.407
  18:16:07 STEP 1999 H=1999:00 +CPU= 20.165
  18:16:28 STEP 2000 H=2000:00 +CPU= 21.545
  18:17:00 STEP 2001 H=2001:00 +CPU= 31.791
  18:17:21 STEP 2002 H=2002:00 +CPU= 20.718
  18:17:42 STEP 2003 H=2003:00 +CPU= 21.006
  18:18:16 STEP 2004 H=2004:00 +CPU= 34.372
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002004
  18:18:37 STEP 2005 H=2005:00 +CPU= 20.677
  18:18:58 STEP 2006 H=2006:00 +CPU= 21.038
  18:19:30 STEP 2007 H=2007:00 +CPU= 31.778
  18:19:51 STEP 2008 H=2008:00 +CPU= 20.912
  18:20:12 STEP 2009 H=2009:00 +CPU= 21.377
  18:20:45 STEP 2010 H=2010:00 +CPU= 32.566
  18:21:05 STEP 2011 H=2011:00 +CPU= 20.610
  18:21:27 STEP 2012 H=2012:00 +CPU= 21.001
  18:21:57 STEP 2013 H=2013:00 +CPU= 30.709
  18:22:17 STEP 2014 H=2014:00 +CPU= 19.918
  18:22:38 STEP 2015 H=2015:00 +CPU= 20.693
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0366
(argv4) batchid: 962
(argv5) wuid: 12179010
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179010
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_96932c7733c61880ef0897837abedb3f to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0366_1993050100_123_962_12179010.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0366_1993050100_123_962_12179010.zip
ic_ancil_file: ic_ancil_12179010
ifsdata_file: ifsdata_12179010
climate_data_file: clim_data_12179010
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_13cb71b4dc22958601b9500812a29942 to: /var/lib/boinc-client/slots/3/ic_ancil_12179010.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12179010.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_88cb4faa6629e0c1a170ece815b6b324 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12179010.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12179010.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12179010.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12179010.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12179010.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12179010.xml
-- Model is restarting --
Adjusting last_iter, 2016, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12179010.xml
last_cpu_time: 51769
upload_file_number: 83
last_iter: 2015
last_upload: 7171200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0366_1993050100_123_962_12179010_0_r1505213178
The child process has been launched with process id: 1401
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182917:1674235757:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
ECCODES ERROR   :  IO ERROR: Inappropriate ioctl for device (Inappropriate ioctl for device)
 GRIB_READ_BYTES_REAL8        50000  FAILED         -11
 GRIB_API ERROR MSG: Input output problem
MPL_ABORT: CALLED FROM PROCESSOR      1 THRD     1
 MPL_ABORT: THRD           1   GRIB_READ_BYTES_REAL8 FAILED
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182922:1674235762:4.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1348 MB (maxheap), 703 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182922:1674235762:4.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182922:1674235762:4.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182922:1674235762:4.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182922:1674235762:4.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182922:1674235762:4.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182922:1674235762:4.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       RERESF 
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182922:1674235762:4.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        IOSTREAM_MIX:IO_GET 
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182922:1674235762:4.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         IOSTREAM_MIX:READ_RECORD 
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182922:1674235762:4.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          >MPL-IOSTREAM READ_RECORD ( 650) 
[EC_DRHOOK:W31Ub20:1:1:1401:1401] [20230120:182922:1674235762:4.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GRIB_API:IGRIB_READ_BYTES_REAL8 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/3/oifs_43r3_model.exe' : sigcontextptr=0x7ffe0a799380
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 16) with plain hex-dump
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [0]: [0x839273] : 0x839273
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [1]: [0x839879] : 0x839879
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [2]: [0x8c2957] : 0x8c2957
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [3]: [0x8c3229] : 0x8c3229
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [4]: [0xa048b9] : 0xa048b9
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [5]: [0x9e3c9a] : 0x9e3c9a
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [6]: [0x9db20d] : 0x9db20d
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [7]: [0x54cfed] : 0x54cfed
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [8]: [0x546a18] : 0x546a18
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [9]: [0x546412] : 0x546412
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [10]: [0x545e28] : 0x545e28
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [11]: [0x40708d] : 0x40708d
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [12]: [0x40220f] : 0x40220f
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [13]: [0x4021a2] : 0x4021a2
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [14]: [0x1dca390] : 0x1dca390
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [15]: [0x40206e] : 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 SDL_TRACEBACK: Done INTEL_TRBK, THRD =            1
abort: 
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:31:06 1532184 22263902 oifs_43r3_ps_0366_1993050100_123_962_12179010_0 6,912,000 49,015 0.0071
09 May 2023 16:30:54 1532184 22263902 oifs_43r3_ps_0366_1993050100_123_962_12179010_0 6,048,000 42,756 0.0071
09 May 2023 16:30:49 1532184 22263902 oifs_43r3_ps_0366_1993050100_123_962_12179010_0 5,184,000 36,508 0.0070
09 May 2023 16:30:37 1532184 22263902 oifs_43r3_ps_0366_1993050100_123_962_12179010_0 4,320,000 30,249 0.0070
09 May 2023 16:30:25 1532184 22263902 oifs_43r3_ps_0366_1993050100_123_962_12179010_0 3,456,000 24,000 0.0069
09 May 2023 16:30:13 1532184 22263902 oifs_43r3_ps_0366_1993050100_123_962_12179010_0 2,592,000 17,771 0.0069
09 May 2023 16:30:06 1532184 22263902 oifs_43r3_ps_0366_1993050100_123_962_12179010_0 1,728,000 11,603 0.0067
09 May 2023 16:29:55 1532184 22263902 oifs_43r3_ps_0366_1993050100_123_962_12179010_0 864,000 5,695 0.0066


©2024 cpdn.org