climateprediction.net home page
Task 22293085

Task 22293085

Name oifs_43r3_ps_0837_2020050100_123_989_12206481_0
Workunit 12206481
Created 25 Dec 2022, 3:01:03 UTC
Sent 20 Jan 2023, 4:21:07 UTC
Report deadline 21 Mar 2023, 4:21:07 UTC
Received 20 Jan 2023, 22:15:16 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1538236
Run time 17 hours 26 min 21 sec
CPU time 17 hours 23 min
Validate state Invalid
Credit 10,886.40
Device peak FLOPS 5.98 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.60 MB
Peak swap size 4,974.28 MB
Peak disk usage 1,221.24 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
+CPU= 38.471
  17:44:29 STEP 1846 H=1846:00 +CPU= 26.171
  17:44:55 STEP 1847 H=1847:00 +CPU= 25.920
  17:45:38 STEP 1848 H=1848:00 +CPU= 43.202
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_76.zip
Uploading the intermediate file: upload_file_76.zip
  17:46:01 STEP 1849 H=1849:00 +CPU= 22.839
  17:46:25 STEP 1850 H=1850:00 +CPU= 23.979
  17:47:03 STEP 1851 H=1851:00 +CPU= 38.510
  17:47:28 STEP 1852 H=1852:00 +CPU= 24.875
  17:47:51 STEP 1853 H=1853:00 +CPU= 22.860
  17:48:30 STEP 1854 H=1854:00 +CPU= 38.856
  17:48:53 STEP 1855 H=1855:00 +CPU= 23.159
  17:49:18 STEP 1856 H=1856:00 +CPU= 25.360
  17:49:56 STEP 1857 H=1857:00 +CPU= 37.485
  17:50:20 STEP 1858 H=1858:00 +CPU= 24.646
  17:50:42 STEP 1859 H=1859:00 +CPU= 21.273
  17:51:24 STEP 1860 H=1860:00 +CPU= 42.566
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001860
  17:51:49 STEP 1861 H=1861:00 +CPU= 24.453
  17:52:15 STEP 1862 H=1862:00 +CPU= 26.028
  17:52:53 STEP 1863 H=1863:00 +CPU= 38.568
  17:53:19 STEP 1864 H=1864:00 +CPU= 25.892
  17:53:45 STEP 1865 H=1865:00 +CPU= 25.270
  17:54:24 STEP 1866 H=1866:00 +CPU= 39.579
  17:54:48 STEP 1867 H=1867:00 +CPU= 24.040
  17:55:13 STEP 1868 H=1868:00 +CPU= 25.191
  17:55:52 STEP 1869 H=1869:00 +CPU= 38.442
  17:56:16 STEP 1870 H=1870:00 +CPU= 23.721
  17:56:42 STEP 1871 H=1871:00 +CPU= 25.949
  17:57:28 STEP 1872 H=1872:00 +CPU= 46.800
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_77.zip
Uploading the intermediate file: upload_file_77.zip
  17:57:53 STEP 1873 H=1873:00 +CPU= 24.360
  17:58:16 STEP 1874 H=1874:00 +CPU= 23.584
  17:58:57 STEP 1875 H=1875:00 +CPU= 40.085
  17:59:23 STEP 1876 H=1876:00 +CPU= 26.518
  17:59:48 STEP 1877 H=1877:00 +CPU= 24.691
  18:00:23 STEP 1878 H=1878:00 +CPU= 35.516
  18:00:48 STEP 1879 H=1879:00 +CPU= 24.975
  18:01:14 STEP 1880 H=1880:00 +CPU= 25.307
  18:01:53 STEP 1881 H=1881:00 +CPU= 39.521
  18:02:20 STEP 1882 H=1882:00 +CPU= 26.734
  18:02:46 STEP 1883 H=1883:00 +CPU= 25.930
  18:03:31 STEP 1884 H=1884:00 +CPU= 45.324
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001884
  18:03:54 STEP 1885 H=1885:00 +CPU= 22.766
  18:04:19 STEP 1886 H=1886:00 +CPU= 24.902
  18:04:58 STEP 1887 H=1887:00 +CPU= 38.668
  18:05:24 STEP 1888 H=1888:00 +CPU= 26.052
  18:05:48 STEP 1889 H=1889:00 +CPU= 23.992
  18:06:27 STEP 1890 H=1890:00 +CPU= 38.717
  18:06:51 STEP 1891 H=1891:00 +CPU= 24.879
  18:07:15 STEP 1892 H=1892:00 +CPU= 23.755
  18:07:53 STEP 1893 H=1893:00 +CPU= 38.198
  18:08:18 STEP 1894 H=1894:00 +CPU= 24.815
  18:08:43 STEP 1895 H=1895:00 +CPU= 25.150
  18:09:29 STEP 1896 H=1896:00 +CPU= 45.125
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_78.zip
Uploading the intermediate file: upload_file_78.zip
  18:09:51 STEP 1897 H=1897:00 +CPU= 22.761
  18:10:16 STEP 1898 H=1898:00 +CPU= 25.009
  18:10:55 STEP 1899 H=1899:00 +CPU= 38.503
  18:11:21 STEP 1900 H=1900:00 +CPU= 25.835
  18:11:46 STEP 1901 H=1901:00 +CPU= 24.722
  18:12:24 STEP 1902 H=1902:00 +CPU= 38.402
  18:12:49 STEP 1903 H=1903:00 +CPU= 24.636
  18:13:13 STEP 1904 H=1904:00 +CPU= 24.298
  18:13:51 STEP 1905 H=1905:00 +CPU= 38.266
  18:14:18 STEP 1906 H=1906:00 +CPU= 26.396
  18:14:43 STEP 1907 H=1907:00 +CPU= 25.454
  18:15:25 STEP 1908 H=1908:00 +CPU= 41.490
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001908
  18:15:49 STEP 1909 H=1909:00 +CPU= 24.416
  18:16:14 STEP 1910 H=1910:00 +CPU= 24.974
  18:16:54 STEP 1911 H=1911:00 +CPU= 39.542
  18:17:21 STEP 1912 H=1912:00 +CPU= 26.967
  18:17:47 STEP 1913 H=1913:00 +CPU= 25.985
  18:18:28 STEP 1914 H=1914:00 +CPU= 41.321
  18:18:54 STEP 1915 H=1915:00 +CPU= 25.604
  18:19:19 STEP 1916 H=1916:00 +CPU= 25.262
  18:19:56 STEP 1917 H=1917:00 +CPU= 37.523
  18:20:22 STEP 1918 H=1918:00 +CPU= 25.779
  18:20:49 STEP 1919 H=1919:00 +CPU= 26.306
  18:21:36 STEP 1920 H=1920:00 +CPU= 47.704
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_79.zip
Uploading the intermediate file: upload_file_79.zip
  18:22:01 STEP 1921 H=1921:00 +CPU= 24.318
Uploading trickle at timestep: 6912000
  18:22:27 STEP 1922 H=1922:00 +CPU= 26.075
  18:23:05 STEP 1923 H=1923:00 +CPU= 38.554
  18:23:31 STEP 1924 H=1924:00 +CPU= 25.993
  18:23:57 STEP 1925 H=1925:00 +CPU= 25.760
  18:24:37 STEP 1926 H=1926:00 +CPU= 39.944
  18:25:02 STEP 1927 H=1927:00 +CPU= 24.426
  18:25:27 STEP 1928 H=1928:00 +CPU= 25.689
  18:26:06 STEP 1929 H=1929:00 +CPU= 39.137
  18:26:32 STEP 1930 H=1930:00 +CPU= 25.399
  18:26:56 STEP 1931 H=1931:00 +CPU= 24.386
  18:27:38 STEP 1932 H=1932:00 +CPU= 41.554
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001932
  18:28:01 STEP 1933 H=1933:00 +CPU= 22.805
  18:28:27 STEP 1934 H=1934:00 +CPU= 26.255
  18:29:06 STEP 1935 H=1935:00 +CPU= 39.152
  18:29:31 STEP 1936 H=1936:00 +CPU= 24.981
  18:29:56 STEP 1937 H=1937:00 +CPU= 24.951
  18:30:35 STEP 1938 H=1938:00 +CPU= 39.223
  18:31:00 STEP 1939 H=1939:00 +CPU= 24.848
  18:31:24 STEP 1940 H=1940:00 +CPU= 23.899
  18:32:04 STEP 1941 H=1941:00 +CPU= 39.385
  18:32:29 STEP 1942 H=1942:00 +CPU= 25.668
  18:32:54 STEP 1943 H=1943:00 +CPU= 24.933
  18:33:40 STEP 1944 H=1944:00 +CPU= 45.865
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_80.zip
Uploading the intermediate file: upload_file_80.zip
  18:34:03 STEP 1945 H=1945:00 +CPU= 23.194
  18:34:28 STEP 1946 H=1946:00 +CPU= 24.942
  18:35:08 STEP 1947 H=1947:00 +CPU= 39.277
  18:35:34 STEP 1948 H=1948:00 +CPU= 25.994
  18:35:58 STEP 1949 H=1949:00 +CPU= 23.974
  18:36:38 STEP 1950 H=1950:00 +CPU= 40.513
  18:37:02 STEP 1951 H=1951:00 +CPU= 23.952
  18:37:27 STEP 1952 H=1952:00 +CPU= 24.577
  18:38:06 STEP 1953 H=1953:00 +CPU= 39.814
  18:38:32 STEP 1954 H=1954:00 +CPU= 25.588
  18:38:55 STEP 1955 H=1955:00 +CPU= 23.351
  18:39:40 STEP 1956 H=1956:00 +CPU= 44.218
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001956
  18:40:05 STEP 1957 H=1957:00 +CPU= 25.179
  18:40:31 STEP 1958 H=1958:00 +CPU= 26.186
  18:41:08 STEP 1959 H=1959:00 +CPU= 37.006
  18:41:31 STEP 1960 H=1960:00 +CPU= 23.055
  18:41:54 STEP 1961 H=1961:00 +CPU= 22.858
  18:42:33 STEP 1962 H=1962:00 +CPU= 39.410
  18:42:57 STEP 1963 H=1963:00 +CPU= 23.270
  18:43:21 STEP 1964 H=1964:00 +CPU= 24.211
  18:44:00 STEP 1965 H=1965:00 +CPU= 38.845
  18:44:23 STEP 1966 H=1966:00 +CPU= 23.176
  18:44:49 STEP 1967 H=1967:00 +CPU= 25.544
  18:45:35 STEP 1968 H=1968:00 +CPU= 46.265
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_81.zip
Uploading the intermediate file: upload_file_81.zip
  18:46:00 STEP 1969 H=1969:00 +CPU= 25.164
  18:46:25 STEP 1970 H=1970:00 +CPU= 25.406
  18:47:05 STEP 1971 H=1971:00 +CPU= 39.178
  18:47:30 STEP 1972 H=1972:00 +CPU= 25.270
  18:47:54 STEP 1973 H=1973:00 +CPU= 24.291
  18:48:32 STEP 1974 H=1974:00 +CPU= 38.001
  18:48:55 STEP 1975 H=1975:00 +CPU= 22.828
  18:49:19 STEP 1976 H=1976:00 +CPU= 24.252
  18:49:57 STEP 1977 H=1977:00 +CPU= 37.889
  18:50:16 STEP 1978 H=1978:00 +CPU= 19.021
  18:50:40 STEP 1979 H=1979:00 +CPU= 23.841
  18:51:22 STEP 1980 H=1980:00 +CPU= 41.763
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001980
  18:51:47 STEP 1981 H=1981:00 +CPU= 24.562
  18:52:13 STEP 1982 H=1982:00 +CPU= 25.878
  18:52:48 STEP 1983 H=1983:00 +CPU= 35.320
  18:53:13 STEP 1984 H=1984:00 +CPU= 25.277
  18:53:38 STEP 1985 H=1985:00 +CPU= 24.522
  18:54:18 STEP 1986 H=1986:00 +CPU= 40.067
  18:54:43 STEP 1987 H=1987:00 +CPU= 25.232
  18:55:10 STEP 1988 H=1988:00 +CPU= 26.421
  18:55:49 STEP 1989 H=1989:00 +CPU= 39.142
  18:56:14 STEP 1990 H=1990:00 +CPU= 25.152
  18:56:40 STEP 1991 H=1991:00 +CPU= 25.560
  18:57:27 STEP 1992 H=1992:00 +CPU= 47.465
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_82.zip
Uploading the intermediate file: upload_file_82.zip
  18:57:53 STEP 1993 H=1993:00 +CPU= 26.005
  18:58:20 STEP 1994 H=1994:00 +CPU= 26.830
  18:59:00 STEP 1995 H=1995:00 +CPU= 40.314
  18:59:24 STEP 1996 H=1996:00 +CPU= 23.916
  18:59:49 STEP 1997 H=1997:00 +CPU= 25.051
  19:00:29 STEP 1998 H=1998:00 +CPU= 39.719
  19:00:54 STEP 1999 H=1999:00 +CPU= 25.037
  19:01:20 STEP 2000 H=2000:00 +CPU= 25.558
  19:01:59 STEP 2001 H=2001:00 +CPU= 39.382
  19:02:24 STEP 2002 H=2002:00 +CPU= 24.773
  19:02:49 STEP 2003 H=2003:00 +CPU= 25.070
  19:03:33 STEP 2004 H=2004:00 +CPU= 43.823
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002004
  19:03:58 STEP 2005 H=2005:00 +CPU= 25.645
  19:04:25 STEP 2006 H=2006:00 +CPU= 26.456
  19:05:04 STEP 2007 H=2007:00 +CPU= 39.366
  19:05:27 STEP 2008 H=2008:00 +CPU= 22.732
  19:05:53 STEP 2009 H=2009:00 +CPU= 25.523
  19:06:34 STEP 2010 H=2010:00 +CPU= 41.123
  19:06:59 STEP 2011 H=2011:00 +CPU= 25.212
  19:07:23 STEP 2012 H=2012:00 +CPU= 24.455
  19:08:04 STEP 2013 H=2013:00 +CPU= 40.788
  19:08:30 STEP 2014 H=2014:00 +CPU= 25.721
  19:08:55 STEP 2015 H=2015:00 +CPU= 25.310
  19:09:39 STEP 2016 H=2016:00 +CPU= 43.843
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_83.zip
Uploading the intermediate file: upload_file_83.zip
  19:10:02 STEP 2017 H=2017:00 +CPU= 22.778
  19:10:28 STEP 2018 H=2018:00 +CPU= 25.968
  19:11:03 STEP 2019 H=2019:00 +CPU= 34.997
  19:11:28 STEP 2020 H=2020:00 +CPU= 25.132
  19:11:54 STEP 2021 H=2021:00 +CPU= 26.027
  19:12:32 STEP 2022 H=2022:00 +CPU= 37.984
  19:12:55 STEP 2023 H=2023:00 +CPU= 23.313
  19:13:20 STEP 2024 H=2024:00 +CPU= 24.928
  19:14:00 STEP 2025 H=2025:00 +CPU= 39.350
  19:14:23 STEP 2026 H=2026:00 +CPU= 23.599
  19:14:48 STEP 2027 H=2027:00 +CPU= 24.129
  19:15:32 STEP 2028 H=2028:00 +CPU= 43.993
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002028
  19:15:57 STEP 2029 H=2029:00 +CPU= 24.905
  19:16:23 STEP 2030 H=2030:00 +CPU= 26.387
  19:17:03 STEP 2031 H=2031:00 +CPU= 39.695
  19:17:27 STEP 2032 H=2032:00 +CPU= 24.693
  19:17:51 STEP 2033 H=2033:00 +CPU= 23.694
  19:18:29 STEP 2034 H=2034:00 +CPU= 37.566
  19:18:53 STEP 2035 H=2035:00 +CPU= 24.862
  19:19:19 STEP 2036 H=2036:00 +CPU= 25.684
  19:19:54 STEP 2037 H=2037:00 +CPU= 35.108
  19:20:18 STEP 2038 H=2038:00 +CPU= 23.228
  19:20:43 STEP 2039 H=2039:00 +CPU= 25.077
  19:21:29 STEP 2040 H=2040:00 +CPU= 45.903
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_84.zip
Uploading the intermediate file: upload_file_84.zip
  19:21:54 STEP 2041 H=2041:00 +CPU= 25.643
  19:22:21 STEP 2042 H=2042:00 +CPU= 26.301
  19:23:01 STEP 2043 H=2043:00 +CPU= 40.088
  19:23:27 STEP 2044 H=2044:00 +CPU= 26.062
  19:23:51 STEP 2045 H=2045:00 +CPU= 24.294
  19:24:28 STEP 2046 H=2046:00 +CPU= 37.342
  19:24:52 STEP 2047 H=2047:00 +CPU= 23.631
  19:25:16 STEP 2048 H=2048:00 +CPU= 24.046
  19:25:54 STEP 2049 H=2049:00 +CPU= 37.544
  19:26:18 STEP 2050 H=2050:00 +CPU= 24.366
  19:26:42 STEP 2051 H=2051:00 +CPU= 23.869
  19:27:19 STEP 2052 H=2052:00 +CPU= 37.270
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002052
  19:27:38 STEP 2053 H=2053:00 +CPU= 19.075
  19:28:00 STEP 2054 H=2054:00 +CPU= 22.139
  19:28:38 STEP 2055 H=2055:00 +CPU= 37.339
  19:29:04 STEP 2056 H=2056:00 +CPU= 25.809
  19:29:28 STEP 2057 H=2057:00 +CPU= 24.332
  19:30:06 STEP 2058 H=2058:00 +CPU= 38.479
  19:30:28 STEP 2059 H=2059:00 +CPU= 21.261
  19:30:54 STEP 2060 H=2060:00 +CPU= 25.881
  19:31:33 STEP 2061 H=2061:00 +CPU= 39.256
  19:31:59 STEP 2062 H=2062:00 +CPU= 25.835
  19:32:25 STEP 2063 H=2063:00 +CPU= 26.500
  19:33:12 STEP 2064 H=2064:00 +CPU= 47.092
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_85.zip
Uploading the intermediate file: upload_file_85.zip
  19:33:37 STEP 2065 H=2065:00 +CPU= 25.147
  19:34:03 STEP 2066 H=2066:00 +CPU= 25.437
  19:34:43 STEP 2067 H=2067:00 +CPU= 39.940
  19:35:09 STEP 2068 H=2068:00 +CPU= 26.076
  19:35:33 STEP 2069 H=2069:00 +CPU= 23.605
  19:36:12 STEP 2070 H=2070:00 +CPU= 39.187
  19:36:37 STEP 2071 H=2071:00 +CPU= 25.028
  19:37:03 STEP 2072 H=2072:00 +CPU= 25.671
  19:37:42 STEP 2073 H=2073:00 +CPU= 39.037
  19:38:07 STEP 2074 H=2074:00 +CPU= 25.618
  19:38:32 STEP 2075 H=2075:00 +CPU= 24.900
  19:39:15 STEP 2076 H=2076:00 +CPU= 42.829
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002076
  19:39:40 STEP 2077 H=2077:00 +CPU= 25.038
  19:40:06 STEP 2078 H=2078:00 +CPU= 26.112
  19:40:44 STEP 2079 H=2079:00 +CPU= 37.951
  19:41:10 STEP 2080 H=2080:00 +CPU= 25.825
  19:41:35 STEP 2081 H=2081:00 +CPU= 25.239
  19:42:14 STEP 2082 H=2082:00 +CPU= 38.803
  19:42:39 STEP 2083 H=2083:00 +CPU= 24.439
  19:43:04 STEP 2084 H=2084:00 +CPU= 25.743
  19:43:44 STEP 2085 H=2085:00 +CPU= 39.627
  19:44:07 STEP 2086 H=2086:00 +CPU= 22.918
  19:44:34 STEP 2087 H=2087:00 +CPU= 26.706
  19:45:18 STEP 2088 H=2088:00 +CPU= 44.382
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_86.zip
Uploading the intermediate file: upload_file_86.zip
  19:45:43 STEP 2089 H=2089:00 +CPU= 24.530
  19:46:08 STEP 2090 H=2090:00 +CPU= 25.670
  19:46:48 STEP 2091 H=2091:00 +CPU= 39.686
  19:47:15 STEP 2092 H=2092:00 +CPU= 26.667
  19:47:41 STEP 2093 H=2093:00 +CPU= 25.964
  19:48:20 STEP 2094 H=2094:00 +CPU= 39.746
  19:48:45 STEP 2095 H=2095:00 +CPU= 24.951
  19:49:08 STEP 2096 H=2096:00 +CPU= 22.685
  19:49:47 STEP 2097 H=2097:00 +CPU= 39.049
  19:50:13 STEP 2098 H=2098:00 +CPU= 25.725
  19:50:37 STEP 2099 H=2099:00 +CPU= 24.625
  19:51:22 STEP 2100 H=2100:00 +CPU= 44.394
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002100
  19:51:47 STEP 2101 H=2101:00 +CPU= 24.773
  19:52:13 STEP 2102 H=2102:00 +CPU= 26.204
  19:52:52 STEP 2103 H=2103:00 +CPU= 39.578
  19:53:19 STEP 2104 H=2104:00 +CPU= 26.382
  19:53:44 STEP 2105 H=2105:00 +CPU= 25.673
  19:54:25 STEP 2106 H=2106:00 +CPU= 40.752
  19:54:51 STEP 2107 H=2107:00 +CPU= 25.720
  19:55:17 STEP 2108 H=2108:00 +CPU= 25.613
  19:55:57 STEP 2109 H=2109:00 +CPU= 39.920
  19:56:23 STEP 2110 H=2110:00 +CPU= 26.062
  19:56:49 STEP 2111 H=2111:00 +CPU= 26.625
  19:57:37 STEP 2112 H=2112:00 +CPU= 47.571
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_87.zip
Uploading the intermediate file: upload_file_87.zip
  19:58:01 STEP 2113 H=2113:00 +CPU= 24.544
  19:58:28 STEP 2114 H=2114:00 +CPU= 26.282
  19:59:07 STEP 2115 H=2115:00 +CPU= 38.895
  19:59:32 STEP 2116 H=2116:00 +CPU= 25.151
  19:59:57 STEP 2117 H=2117:00 +CPU= 25.412
  20:00:36 STEP 2118 H=2118:00 +CPU= 38.685
  20:01:00 STEP 2119 H=2119:00 +CPU= 24.730
  20:01:27 STEP 2120 H=2120:00 +CPU= 26.305
  20:02:04 STEP 2121 H=2121:00 +CPU= 37.542
  20:02:28 STEP 2122 H=2122:00 +CPU= 23.722
  20:02:53 STEP 2123 H=2123:00 +CPU= 25.373
  20:03:38 STEP 2124 H=2124:00 +CPU= 44.534
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002124
  20:04:03 STEP 2125 H=2125:00 +CPU= 25.431
  20:04:30 STEP 2126 H=2126:00 +CPU= 26.706
  20:05:10 STEP 2127 H=2127:00 +CPU= 39.529
  20:05:35 STEP 2128 H=2128:00 +CPU= 25.423
  20:06:00 STEP 2129 H=2129:00 +CPU= 24.841
  20:06:40 STEP 2130 H=2130:00 +CPU= 39.675
  20:07:04 STEP 2131 H=2131:00 +CPU= 24.235
  20:07:29 STEP 2132 H=2132:00 +CPU= 24.883
  20:08:08 STEP 2133 H=2133:00 +CPU= 39.021
  20:08:34 STEP 2134 H=2134:00 +CPU= 26.330
  20:09:00 STEP 2135 H=2135:00 +CPU= 25.706
  20:09:46 STEP 2136 H=2136:00 +CPU= 46.079
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_88.zip
Uploading the intermediate file: upload_file_88.zip
  20:10:11 STEP 2137 H=2137:00 +CPU= 24.690
  20:10:36 STEP 2138 H=2138:00 +CPU= 25.087
  20:11:13 STEP 2139 H=2139:00 +CPU= 37.405
  20:11:38 STEP 2140 H=2140:00 +CPU= 24.588
  20:12:03 STEP 2141 H=2141:00 +CPU= 24.760
  20:12:39 STEP 2142 H=2142:00 +CPU= 36.129
  20:13:03 STEP 2143 H=2143:00 +CPU= 24.465
  20:13:28 STEP 2144 H=2144:00 +CPU= 24.927
  20:14:07 STEP 2145 H=2145:00 +CPU= 38.532
  20:14:33 STEP 2146 H=2146:00 +CPU= 26.325
  20:14:57 STEP 2147 H=2147:00 +CPU= 24.156
  20:15:38 STEP 2148 H=2148:00 +CPU= 40.258
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002148
  20:15:59 STEP 2149 H=2149:00 +CPU= 21.714
  20:16:22 STEP 2150 H=2150:00 +CPU= 22.345
  20:16:59 STEP 2151 H=2151:00 +CPU= 36.945
  20:17:23 STEP 2152 H=2152:00 +CPU= 24.218
  20:17:48 STEP 2153 H=2153:00 +CPU= 25.056
  20:18:26 STEP 2154 H=2154:00 +CPU= 37.681
  20:18:51 STEP 2155 H=2155:00 +CPU= 24.776
  20:19:14 STEP 2156 H=2156:00 +CPU= 23.750
  20:19:51 STEP 2157 H=2157:00 +CPU= 36.809
  20:20:13 STEP 2158 H=2158:00 +CPU= 21.804
  20:20:35 STEP 2159 H=2159:00 +CPU= 22.549
  20:21:17 STEP 2160 H=2160:00 +CPU= 41.185
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_89.zip
Uploading the intermediate file: upload_file_89.zip
Uploading trickle at timestep: 7776000
  20:21:39 STEP 2161 H=2161:00 +CPU= 21.939
  20:22:03 STEP 2162 H=2162:00 +CPU= 23.935
  20:22:39 STEP 2163 H=2163:00 +CPU= 36.494
  20:23:03 STEP 2164 H=2164:00 +CPU= 23.527
  20:23:28 STEP 2165 H=2165:00 +CPU= 25.452
  20:24:08 STEP 2166 H=2166:00 +CPU= 40.290
  20:24:33 STEP 2167 H=2167:00 +CPU= 24.699
  20:24:58 STEP 2168 H=2168:00 +CPU= 24.468
  20:25:36 STEP 2169 H=2169:00 +CPU= 38.466
  20:26:01 STEP 2170 H=2170:00 +CPU= 24.751
  20:26:26 STEP 2171 H=2171:00 +CPU= 25.194
  20:27:07 STEP 2172 H=2172:00 +CPU= 41.446
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002172
  20:27:29 STEP 2173 H=2173:00 +CPU= 21.792
  20:27:54 STEP 2174 H=2174:00 +CPU= 24.527
  20:28:33 STEP 2175 H=2175:00 +CPU= 39.399
  20:28:58 STEP 2176 H=2176:00 +CPU= 24.597
  20:29:22 STEP 2177 H=2177:00 +CPU= 24.207
  20:30:03 STEP 2178 H=2178:00 +CPU= 40.642
  20:30:28 STEP 2179 H=2179:00 +CPU= 24.785
  20:30:53 STEP 2180 H=2180:00 +CPU= 25.272
  20:31:31 STEP 2181 H=2181:00 +CPU= 38.162
  20:31:55 STEP 2182 H=2182:00 +CPU= 23.761
  20:32:19 STEP 2183 H=2183:00 +CPU= 23.634
  20:33:04 STEP 2184 H=2184:00 +CPU= 45.569
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_90.zip
Uploading the intermediate file: upload_file_90.zip
  20:33:29 STEP 2185 H=2185:00 +CPU= 25.122
  20:33:55 STEP 2186 H=2186:00 +CPU= 25.814
  20:34:35 STEP 2187 H=2187:00 +CPU= 40.315
  20:35:00 STEP 2188 H=2188:00 +CPU= 25.000
  20:35:24 STEP 2189 H=2189:00 +CPU= 23.061
  20:36:01 STEP 2190 H=2190:00 +CPU= 37.168
  20:36:24 STEP 2191 H=2191:00 +CPU= 23.392
  20:36:50 STEP 2192 H=2192:00 +CPU= 25.829
  20:37:27 STEP 2193 H=2193:00 +CPU= 37.313
  20:37:54 STEP 2194 H=2194:00 +CPU= 26.645
  20:38:19 STEP 2195 H=2195:00 +CPU= 24.485
  20:39:04 STEP 2196 H=2196:00 +CPU= 45.426
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002196
  20:39:30 STEP 2197 H=2197:00 +CPU= 26.108
  20:39:58 STEP 2198 H=2198:00 +CPU= 27.326
  20:40:40 STEP 2199 H=2199:00 +CPU= 42.463
  20:41:07 STEP 2200 H=2200:00 +CPU= 26.563
  20:41:34 STEP 2201 H=2201:00 +CPU= 27.743
  20:42:17 STEP 2202 H=2202:00 +CPU= 42.692
  20:42:43 STEP 2203 H=2203:00 +CPU= 26.035
  20:43:10 STEP 2204 H=2204:00 +CPU= 27.270
  20:43:52 STEP 2205 H=2205:00 +CPU= 41.173
  20:44:19 STEP 2206 H=2206:00 +CPU= 27.269
  20:44:43 STEP 2207 H=2207:00 +CPU= 24.212
  20:45:29 STEP 2208 H=2208:00 +CPU= 45.980
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_91.zip
Uploading the intermediate file: upload_file_91.zip
  20:45:55 STEP 2209 H=2209:00 +CPU= 26.376
  20:46:21 STEP 2210 H=2210:00 +CPU= 25.251
  20:47:02 STEP 2211 H=2211:00 +CPU= 41.186
  20:47:28 STEP 2212 H=2212:00 +CPU= 26.188
  20:47:54 STEP 2213 H=2213:00 +CPU= 26.304
  20:48:35 STEP 2214 H=2214:00 +CPU= 40.594
  20:48:59 STEP 2215 H=2215:00 +CPU= 23.677
  20:49:22 STEP 2216 H=2216:00 +CPU= 23.137
  20:50:03 STEP 2217 H=2217:00 +CPU= 41.180
  20:50:30 STEP 2218 H=2218:00 +CPU= 26.704
  20:50:56 STEP 2219 H=2219:00 +CPU= 26.563
  20:51:41 STEP 2220 H=2220:00 +CPU= 44.260
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002220
  20:52:03 STEP 2221 H=2221:00 +CPU= 21.892
  20:52:30 STEP 2222 H=2222:00 +CPU= 27.380
  20:53:12 STEP 2223 H=2223:00 +CPU= 42.195
  20:53:40 STEP 2224 H=2224:00 +CPU= 27.699
  20:54:05 STEP 2225 H=2225:00 +CPU= 25.516
  20:54:44 STEP 2226 H=2226:00 +CPU= 38.280
  20:55:09 STEP 2227 H=2227:00 +CPU= 25.520
  20:55:37 STEP 2228 H=2228:00 +CPU= 27.657
  20:56:17 STEP 2229 H=2229:00 +CPU= 39.674
  20:56:44 STEP 2230 H=2230:00 +CPU= 27.385
  20:57:12 STEP 2231 H=2231:00 +CPU= 27.828
  20:57:59 STEP 2232 H=2232:00 +CPU= 46.881
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_92.zip
Uploading the intermediate file: upload_file_92.zip
  20:58:23 STEP 2233 H=2233:00 +CPU= 24.432
  20:58:50 STEP 2234 H=2234:00 +CPU= 27.045
  20:59:28 STEP 2235 H=2235:00 +CPU= 37.856
  20:59:56 STEP 2236 H=2236:00 +CPU= 27.620
  21:00:21 STEP 2237 H=2237:00 +CPU= 24.923
  21:01:01 STEP 2238 H=2238:00 +CPU= 40.608
  21:01:28 STEP 2239 H=2239:00 +CPU= 26.125
  21:01:55 STEP 2240 H=2240:00 +CPU= 27.051
  21:02:35 STEP 2241 H=2241:00 +CPU= 40.441
  21:03:02 STEP 2242 H=2242:00 +CPU= 26.851
  21:03:27 STEP 2243 H=2243:00 +CPU= 25.134
  21:04:13 STEP 2244 H=2244:00 +CPU= 45.914
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002244
  21:04:40 STEP 2245 H=2245:00 +CPU= 26.864
  21:05:06 STEP 2246 H=2246:00 +CPU= 26.586
  21:05:47 STEP 2247 H=2247:00 +CPU= 40.532
  21:06:13 STEP 2248 H=2248:00 +CPU= 26.185
  21:06:38 STEP 2249 H=2249:00 +CPU= 25.019
  21:07:14 STEP 2250 H=2250:00 +CPU= 36.155
  21:07:40 STEP 2251 H=2251:00 +CPU= 25.324
  21:08:07 STEP 2252 H=2252:00 +CPU= 27.431
  21:08:47 STEP 2253 H=2253:00 +CPU= 39.839
  21:09:12 STEP 2254 H=2254:00 +CPU= 25.279
  21:09:39 STEP 2255 H=2255:00 +CPU= 26.084
  21:10:25 STEP 2256 H=2256:00 +CPU= 46.435
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_93.zip
Uploading the intermediate file: upload_file_93.zip
  21:10:49 STEP 2257 H=2257:00 +CPU= 24.318
  21:11:15 STEP 2258 H=2258:00 +CPU= 25.628
  21:11:55 STEP 2259 H=2259:00 +CPU= 39.394
  21:12:20 STEP 2260 H=2260:00 +CPU= 25.891
  21:12:46 STEP 2261 H=2261:00 +CPU= 25.532
  21:13:27 STEP 2262 H=2262:00 +CPU= 40.693
  21:13:53 STEP 2263 H=2263:00 +CPU= 26.600
  21:14:21 STEP 2264 H=2264:00 +CPU= 27.736
  21:15:03 STEP 2265 H=2265:00 +CPU= 41.511
  21:15:30 STEP 2266 H=2266:00 +CPU= 27.259
  21:15:56 STEP 2267 H=2267:00 +CPU= 26.554
  21:16:40 STEP 2268 H=2268:00 +CPU= 43.176
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002268
  21:17:05 STEP 2269 H=2269:00 +CPU= 25.438
  21:17:31 STEP 2270 H=2270:00 +CPU= 26.276
  21:18:10 STEP 2271 H=2271:00 +CPU= 38.274
  21:18:37 STEP 2272 H=2272:00 +CPU= 26.929
  21:19:03 STEP 2273 H=2273:00 +CPU= 26.403
  21:19:45 STEP 2274 H=2274:00 +CPU= 41.674
  21:20:11 STEP 2275 H=2275:00 +CPU= 26.673
  21:20:37 STEP 2276 H=2276:00 +CPU= 25.000
  21:21:16 STEP 2277 H=2277:00 +CPU= 39.146
  21:21:42 STEP 2278 H=2278:00 +CPU= 25.890
  21:22:06 STEP 2279 H=2279:00 +CPU= 24.728
  21:22:53 STEP 2280 H=2280:00 +CPU= 46.931
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_94.zip
Uploading the intermediate file: upload_file_94.zip
  21:23:19 STEP 2281 H=2281:00 +CPU= 26.176
  21:23:46 STEP 2282 H=2282:00 +CPU= 26.668
  21:24:27 STEP 2283 H=2283:00 +CPU= 40.744
  21:24:49 STEP 2284 H=2284:00 +CPU= 22.452
  21:25:14 STEP 2285 H=2285:00 +CPU= 24.451
  21:25:56 STEP 2286 H=2286:00 +CPU= 41.779
  21:26:22 STEP 2287 H=2287:00 +CPU= 25.991
  21:26:48 STEP 2288 H=2288:00 +CPU= 26.404
  21:27:29 STEP 2289 H=2289:00 +CPU= 40.768
  21:27:56 STEP 2290 H=2290:00 +CPU= 27.151
  21:28:20 STEP 2291 H=2291:00 +CPU= 23.744
  21:29:03 STEP 2292 H=2292:00 +CPU= 42.830
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002292
  21:29:28 STEP 2293 H=2293:00 +CPU= 25.212
  21:29:55 STEP 2294 H=2294:00 +CPU= 26.793
  21:30:32 STEP 2295 H=2295:00 +CPU= 37.543
  21:30:57 STEP 2296 H=2296:00 +CPU= 24.942
  21:31:21 STEP 2297 H=2297:00 +CPU= 24.040
  21:31:58 STEP 2298 H=2298:00 +CPU= 36.812
  21:32:22 STEP 2299 H=2299:00 +CPU= 24.249
  21:32:48 STEP 2300 H=2300:00 +CPU= 26.177
  21:33:25 STEP 2301 H=2301:00 +CPU= 36.038
  21:33:48 STEP 2302 H=2302:00 +CPU= 23.576
  21:34:10 STEP 2303 H=2303:00 +CPU= 21.946
  21:34:53 STEP 2304 H=2304:00 +CPU= 43.259
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150_95.zip
Uploading the intermediate file: upload_file_95.zip
  21:35:16 STEP 2305 H=2305:00 +CPU= 22.668
  21:35:38 STEP 2306 H=2306:00 +CPU= 22.102
  21:36:12 STEP 2307 H=2307:00 +CPU= 34.117
  21:36:34 STEP 2308 H=2308:00 +CPU= 22.169
  21:36:55 STEP 2309 H=2309:00 +CPU= 20.227
  21:37:27 STEP 2310 H=2310:00 +CPU= 32.679
  21:37:48 STEP 2311 H=2311:00 +CPU= 20.734
  21:38:10 STEP 2312 H=2312:00 +CPU= 21.504
  21:38:43 STEP 2313 H=2313:00 +CPU= 33.273
  21:39:03 STEP 2314 H=2314:00 +CPU= 20.411
  21:39:26 STEP 2315 H=2315:00 +CPU= 23.175
  21:40:15 STEP 2316 H=2316:00 +CPU= 48.488
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client_40000/slots/14/ICMUAhq0f+002316
  21:40:41 STEP 2317 H=2317:00 +CPU= 25.991
  21:41:05 STEP 2318 H=2318:00 +CPU= 23.550
  21:41:39 STEP 2319 H=2319:00 +CPU= 34.652
  21:42:03 STEP 2320 H=2320:00 +CPU= 23.976
  21:42:24 STEP 2321 H=2321:00 +CPU= 21.221
  21:42:59 STEP 2322 H=2322:00 +CPU= 34.760
  21:43:21 STEP 2323 H=2323:00 +CPU= 21.540
  21:43:43 STEP 2324 H=2324:00 +CPU= 22.173
  21:44:16 STEP 2325 H=2325:00 +CPU= 33.131
  21:44:39 STEP 2326 H=2326:00 +CPU= 23.332
  21:45:03 STEP 2327 H=2327:00 +CPU= 23.508
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0837
(argv4) batchid: 989
(argv5) wuid: 12206481
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/14
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12206481
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5958e030ed7ff5f27ac17a95bce59e94 to: /var/lib/boinc-client_40000/slots/14/oifs_43r3_ps_0837_2020050100_123_989_12206481.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/14/oifs_43r3_ps_0837_2020050100_123_989_12206481.zip
ic_ancil_file: ic_ancil_12206481
ifsdata_file: ifsdata_12206481
climate_data_file: clim_data_12206481
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4f1322997171568577ac1120c75bdd9e to: /var/lib/boinc-client_40000/slots/14/ic_ancil_12206481.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/14/ic_ancil_12206481.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3b542ffeca6921527a3be2de986196de to: /var/lib/boinc-client_40000/slots/14/ifsdata/ifsdata_12206481.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/14/ifsdata/ifsdata_12206481.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/14/159l_2/clim_data_12206481.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/14/159l_2/clim_data_12206481.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/14/progress_file_12206481.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/14/progress_file_12206481.xml
-- Model is restarting --
Adjusting last_iter, 2328, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/14/progress_file_12206481.xml
last_cpu_time: 62579
upload_file_number: 96
last_iter: 2327
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0837_2020050100_123_989_12206481_0_r1818603150
The child process has been launched with process id: 1429
Executing the command: /var/lib/boinc-client_40000/slots/14/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215828:1674251908:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
ECCODES ERROR   :  IO ERROR: Inappropriate ioctl for device (Inappropriate ioctl for device)
 GRIB_READ_BYTES_REAL8        50000  FAILED         -11
 GRIB_API ERROR MSG: Input output problem
MPL_ABORT: CALLED FROM PROCESSOR      1 THRD     1
 MPL_ABORT: THRD           1   GRIB_READ_BYTES_REAL8 FAILED
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215933:1674251973:65.162] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1504 MB (maxheap), 900 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215933:1674251973:65.162] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215933:1674251973:65.162] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215933:1674251973:65.162] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215933:1674251973:65.162] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215933:1674251973:65.162] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215933:1674251973:65.162] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       RERESF 
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215933:1674251973:65.162] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        IOSTREAM_MIX:IO_GET 
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215933:1674251973:65.162] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         IOSTREAM_MIX:READ_RECORD 
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215933:1674251973:65.162] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          >MPL-IOSTREAM READ_RECORD ( 650) 
[EC_DRHOOK:e1:1:1:1429:1429] [20230120:215933:1674251973:65.162] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GRIB_API:IGRIB_READ_BYTES_REAL8 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client_40000/slots/14/oifs_43r3_model.exe' : sigcontextptr=0x7fff9392ec80
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 16) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client_40000/slots/14/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x8c3229 0xa048b9 0x9e3c9a 0x9da7b4 0x54d557 0x546a18 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x8c3229] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0xa048b9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x9e3c9a] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x9da7b4] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x54d557] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x546a18] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x546412] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x545e28] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x40708d] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x40220f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x4021a2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x1dca390] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40206e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 SDL_TRACEBACK: Done INTEL_TRBK, THRD =            1
abort: 
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:31:38 1538236 22293085 oifs_43r3_ps_0837_2020050100_123_989_12206481_0 7,776,000 57,548 0.0074
09 May 2023 16:31:25 1538236 22293085 oifs_43r3_ps_0837_2020050100_123_989_12206481_0 6,912,000 50,379 0.0073
09 May 2023 16:31:12 1538236 22293085 oifs_43r3_ps_0837_2020050100_123_989_12206481_0 6,048,000 43,179 0.0071
09 May 2023 16:31:07 1538236 22293085 oifs_43r3_ps_0837_2020050100_123_989_12206481_0 5,184,000 36,400 0.0070
09 May 2023 16:30:55 1538236 22293085 oifs_43r3_ps_0837_2020050100_123_989_12206481_0 4,320,000 30,304 0.0070
09 May 2023 16:30:44 1538236 22293085 oifs_43r3_ps_0837_2020050100_123_989_12206481_0 3,456,000 24,300 0.0070
09 May 2023 16:30:38 1538236 22293085 oifs_43r3_ps_0837_2020050100_123_989_12206481_0 2,592,000 18,237 0.0070
09 May 2023 16:30:26 1538236 22293085 oifs_43r3_ps_0837_2020050100_123_989_12206481_0 1,728,000 12,253 0.0071
09 May 2023 16:30:14 1538236 22293085 oifs_43r3_ps_0837_2020050100_123_989_12206481_0 864,000 6,270 0.0073


©2024 cpdn.org