climateprediction.net home page
Task 22304010

Task 22304010

Name oifs_43r3_ps_0298_1998050100_123_967_12183942_2
Workunit 12183942
Created 26 Jan 2023, 19:35:07 UTC
Sent 26 Jan 2023, 19:35:21 UTC
Report deadline 27 Mar 2023, 19:35:21 UTC
Received 30 Jan 2023, 18:03:23 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1511650
Run time 2 days 19 hours 10 min 29 sec
CPU time 2 days 9 hours 23 min 11 sec
Validate state Invalid
Credit 10,160.64
Device peak FLOPS 2.42 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.02 MB
Peak swap size 4,978.14 MB
Peak disk usage 7,688.51 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
p: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_75.zip
Uploading the intermediate file: upload_file_75.zip
  09:20:58 STEP 1825 H=1825:00 +CPU= 69.694
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:23:37 STEP 1826 H=1826:00 +CPU= 72.400
  09:25:46 STEP 1827 H=1827:00 +CPU=113.656
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:27:23 STEP 1828 H=1828:00 +CPU= 72.720
  09:28:46 STEP 1829 H=1829:00 +CPU= 70.106
  09:31:04 STEP 1830 H=1830:00 +CPU=116.892
  09:32:26 STEP 1831 H=1831:00 +CPU= 70.084
  09:33:49 STEP 1832 H=1832:00 +CPU= 72.581
  09:36:02 STEP 1833 H=1833:00 +CPU=113.596
  09:37:26 STEP 1834 H=1834:00 +CPU= 72.544
  09:38:47 STEP 1835 H=1835:00 +CPU= 69.959
  09:41:16 STEP 1836 H=1836:00 +CPU=126.694
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001836
  09:42:35 STEP 1837 H=1837:00 +CPU= 69.638
  09:43:56 STEP 1838 H=1838:00 +CPU= 72.697
  09:46:07 STEP 1839 H=1839:00 +CPU=113.562
  09:47:31 STEP 1840 H=1840:00 +CPU= 73.173
  09:48:50 STEP 1841 H=1841:00 +CPU= 69.906
  09:51:05 STEP 1842 H=1842:00 +CPU=117.082
  09:52:24 STEP 1843 H=1843:00 +CPU= 69.868
  09:53:49 STEP 1844 H=1844:00 +CPU= 72.751
  09:55:59 STEP 1845 H=1845:00 +CPU=113.409
  09:57:20 STEP 1846 H=1846:00 +CPU= 72.292
  09:58:44 STEP 1847 H=1847:00 +CPU= 72.724
  10:01:17 STEP 1848 H=1848:00 +CPU=132.807
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_76.zip
Uploading the intermediate file: upload_file_76.zip
  10:02:38 STEP 1849 H=1849:00 +CPU= 69.966
  10:04:02 STEP 1850 H=1850:00 +CPU= 73.241
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:06:31 STEP 1851 H=1851:00 +CPU=113.688
  10:07:53 STEP 1852 H=1852:00 +CPU= 72.541
  10:09:15 STEP 1853 H=1853:00 +CPU= 70.135
  10:11:29 STEP 1854 H=1854:00 +CPU=116.746
  10:12:50 STEP 1855 H=1855:00 +CPU= 69.725
  10:14:13 STEP 1856 H=1856:00 +CPU= 72.869
  10:16:21 STEP 1857 H=1857:00 +CPU=113.552
  10:17:45 STEP 1858 H=1858:00 +CPU= 72.745
  10:19:07 STEP 1859 H=1859:00 +CPU= 70.495
  10:21:33 STEP 1860 H=1860:00 +CPU=127.201
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001860
  10:22:55 STEP 1861 H=1861:00 +CPU= 70.054
  10:24:20 STEP 1862 H=1862:00 +CPU= 72.611
  10:26:33 STEP 1863 H=1863:00 +CPU=113.724
  10:27:57 STEP 1864 H=1864:00 +CPU= 72.783
  10:29:19 STEP 1865 H=1865:00 +CPU= 69.662
  10:31:32 STEP 1866 H=1866:00 +CPU=116.504
  10:32:53 STEP 1867 H=1867:00 +CPU= 69.884
  10:34:16 STEP 1868 H=1868:00 +CPU= 72.205
  10:36:30 STEP 1869 H=1869:00 +CPU=113.470
  10:37:53 STEP 1870 H=1870:00 +CPU= 73.028
  10:39:16 STEP 1871 H=1871:00 +CPU= 72.569
  10:41:52 STEP 1872 H=1872:00 +CPU=132.997
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_77.zip
Uploading the intermediate file: upload_file_77.zip
  10:43:13 STEP 1873 H=1873:00 +CPU= 69.697
  10:44:37 STEP 1874 H=1874:00 +CPU= 72.792
  10:46:49 STEP 1875 H=1875:00 +CPU=113.389
  10:48:12 STEP 1876 H=1876:00 +CPU= 72.321
  10:49:33 STEP 1877 H=1877:00 +CPU= 70.134
  10:51:45 STEP 1878 H=1878:00 +CPU=117.070
  10:53:04 STEP 1879 H=1879:00 +CPU= 69.879
  10:54:30 STEP 1880 H=1880:00 +CPU= 73.116
  10:56:42 STEP 1881 H=1881:00 +CPU=115.091
  10:58:04 STEP 1882 H=1882:00 +CPU= 72.600
  10:59:27 STEP 1883 H=1883:00 +CPU= 70.247
  11:01:56 STEP 1884 H=1884:00 +CPU=126.718
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001884
  11:03:18 STEP 1885 H=1885:00 +CPU= 70.396
  11:04:41 STEP 1886 H=1886:00 +CPU= 72.352
  11:06:52 STEP 1887 H=1887:00 +CPU=113.336
  11:08:13 STEP 1888 H=1888:00 +CPU= 72.725
  11:09:35 STEP 1889 H=1889:00 +CPU= 69.990
  11:11:51 STEP 1890 H=1890:00 +CPU=117.368
  11:13:13 STEP 1891 H=1891:00 +CPU= 69.750
  11:14:40 STEP 1892 H=1892:00 +CPU= 73.032
  11:16:51 STEP 1893 H=1893:00 +CPU=113.431
  11:18:13 STEP 1894 H=1894:00 +CPU= 72.336
  11:19:38 STEP 1895 H=1895:00 +CPU= 72.912
  11:22:12 STEP 1896 H=1896:00 +CPU=132.102
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_78.zip
Uploading the intermediate file: upload_file_78.zip
  11:23:34 STEP 1897 H=1897:00 +CPU= 69.953
  11:24:57 STEP 1898 H=1898:00 +CPU= 72.818
  11:27:10 STEP 1899 H=1899:00 +CPU=113.136
  11:28:35 STEP 1900 H=1900:00 +CPU= 72.391
  11:29:55 STEP 1901 H=1901:00 +CPU= 70.146
  11:32:11 STEP 1902 H=1902:00 +CPU=117.608
  11:33:31 STEP 1903 H=1903:00 +CPU= 70.070
  11:34:58 STEP 1904 H=1904:00 +CPU= 72.766
  11:37:09 STEP 1905 H=1905:00 +CPU=113.618
  11:38:35 STEP 1906 H=1906:00 +CPU= 72.863
  11:39:56 STEP 1907 H=1907:00 +CPU= 70.211
  11:42:24 STEP 1908 H=1908:00 +CPU=126.616
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001908
  11:43:46 STEP 1909 H=1909:00 +CPU= 70.452
  11:45:11 STEP 1910 H=1910:00 +CPU= 72.770
  11:47:23 STEP 1911 H=1911:00 +CPU=113.836
  11:48:47 STEP 1912 H=1912:00 +CPU= 72.911
  11:50:10 STEP 1913 H=1913:00 +CPU= 69.471
  11:52:24 STEP 1914 H=1914:00 +CPU=116.473
  11:53:47 STEP 1915 H=1915:00 +CPU= 69.924
  11:55:11 STEP 1916 H=1916:00 +CPU= 72.067
  11:57:22 STEP 1917 H=1917:00 +CPU=112.794
  11:58:46 STEP 1918 H=1918:00 +CPU= 72.685
  12:00:10 STEP 1919 H=1919:00 +CPU= 72.543
  12:02:46 STEP 1920 H=1920:00 +CPU=132.821
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_79.zip
Uploading the intermediate file: upload_file_79.zip
  12:04:10 STEP 1921 H=1921:00 +CPU= 70.307
  12:05:35 STEP 1922 H=1922:00 +CPU= 72.240
  12:07:45 STEP 1923 H=1923:00 +CPU=113.539
  12:09:11 STEP 1924 H=1924:00 +CPU= 72.895
  12:10:32 STEP 1925 H=1925:00 +CPU= 69.374
  12:12:46 STEP 1926 H=1926:00 +CPU=116.419
  12:14:07 STEP 1927 H=1927:00 +CPU= 70.140
  12:15:29 STEP 1928 H=1928:00 +CPU= 72.037
  12:17:38 STEP 1929 H=1929:00 +CPU=113.662
  12:19:02 STEP 1930 H=1930:00 +CPU= 73.169
  12:20:24 STEP 1931 H=1931:00 +CPU= 70.103
  12:22:52 STEP 1932 H=1932:00 +CPU=126.984
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001932
  12:24:13 STEP 1933 H=1933:00 +CPU= 69.937
  12:25:38 STEP 1934 H=1934:00 +CPU= 72.787
  12:27:50 STEP 1935 H=1935:00 +CPU=113.659
  12:29:14 STEP 1936 H=1936:00 +CPU= 72.138
  12:30:33 STEP 1937 H=1937:00 +CPU= 70.078
  12:32:45 STEP 1938 H=1938:00 +CPU=116.390
  12:34:08 STEP 1939 H=1939:00 +CPU= 69.816
  12:35:34 STEP 1940 H=1940:00 +CPU= 72.924
  12:37:48 STEP 1941 H=1941:00 +CPU=114.253
  12:39:11 STEP 1942 H=1942:00 +CPU= 72.181
  12:40:37 STEP 1943 H=1943:00 +CPU= 72.772
  12:43:12 STEP 1944 H=1944:00 +CPU=133.153
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_80.zip
Uploading the intermediate file: upload_file_80.zip
  12:44:34 STEP 1945 H=1945:00 +CPU= 69.649
  12:45:59 STEP 1946 H=1946:00 +CPU= 72.691
  12:48:10 STEP 1947 H=1947:00 +CPU=113.715
  12:49:34 STEP 1948 H=1948:00 +CPU= 72.628
  12:50:52 STEP 1949 H=1949:00 +CPU= 69.696
  12:53:09 STEP 1950 H=1950:00 +CPU=116.541
  12:54:31 STEP 1951 H=1951:00 +CPU= 70.183
  12:55:55 STEP 1952 H=1952:00 +CPU= 73.033
  12:58:09 STEP 1953 H=1953:00 +CPU=113.018
  12:59:31 STEP 1954 H=1954:00 +CPU= 72.974
  13:00:52 STEP 1955 H=1955:00 +CPU= 69.799
  13:03:17 STEP 1956 H=1956:00 +CPU=126.630
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001956
  13:04:40 STEP 1957 H=1957:00 +CPU= 69.546
  13:06:05 STEP 1958 H=1958:00 +CPU= 72.749
  13:08:18 STEP 1959 H=1959:00 +CPU=113.773
  13:09:43 STEP 1960 H=1960:00 +CPU= 72.968
  13:11:03 STEP 1961 H=1961:00 +CPU= 69.862
  13:13:17 STEP 1962 H=1962:00 +CPU=116.816
  13:14:36 STEP 1963 H=1963:00 +CPU= 69.616
  13:16:00 STEP 1964 H=1964:00 +CPU= 72.802
  13:18:13 STEP 1965 H=1965:00 +CPU=114.174
  13:19:40 STEP 1966 H=1966:00 +CPU= 72.309
  13:21:05 STEP 1967 H=1967:00 +CPU= 72.690
  13:23:38 STEP 1968 H=1968:00 +CPU=133.307
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_81.zip
Uploading the intermediate file: upload_file_81.zip
  13:24:59 STEP 1969 H=1969:00 +CPU= 69.325
  13:26:25 STEP 1970 H=1970:00 +CPU= 72.978
  13:28:36 STEP 1971 H=1971:00 +CPU=113.552
  13:30:01 STEP 1972 H=1972:00 +CPU= 72.274
  13:31:22 STEP 1973 H=1973:00 +CPU= 70.008
  13:33:36 STEP 1974 H=1974:00 +CPU=116.317
  13:34:58 STEP 1975 H=1975:00 +CPU= 69.314
  13:36:23 STEP 1976 H=1976:00 +CPU= 72.500
  13:38:34 STEP 1977 H=1977:00 +CPU=113.589
  13:39:55 STEP 1978 H=1978:00 +CPU= 72.574
  13:41:17 STEP 1979 H=1979:00 +CPU= 70.383
  13:43:46 STEP 1980 H=1980:00 +CPU=126.342
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001980
  13:45:08 STEP 1981 H=1981:00 +CPU= 69.830
  13:46:33 STEP 1982 H=1982:00 +CPU= 73.169
  13:48:47 STEP 1983 H=1983:00 +CPU=113.557
  13:50:14 STEP 1984 H=1984:00 +CPU= 72.770
  13:51:37 STEP 1985 H=1985:00 +CPU= 70.245
  13:53:54 STEP 1986 H=1986:00 +CPU=116.860
  13:55:13 STEP 1987 H=1987:00 +CPU= 69.581
  13:56:38 STEP 1988 H=1988:00 +CPU= 72.819
  13:58:45 STEP 1989 H=1989:00 +CPU=112.989
  14:00:09 STEP 1990 H=1990:00 +CPU= 72.929
  14:01:34 STEP 1991 H=1991:00 +CPU= 72.662
  14:04:09 STEP 1992 H=1992:00 +CPU=132.371
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_82.zip
Uploading the intermediate file: upload_file_82.zip
  14:05:32 STEP 1993 H=1993:00 +CPU= 69.837
  14:06:55 STEP 1994 H=1994:00 +CPU= 72.344
  14:09:06 STEP 1995 H=1995:00 +CPU=112.714
  14:10:27 STEP 1996 H=1996:00 +CPU= 72.301
  14:11:47 STEP 1997 H=1997:00 +CPU= 69.755
  14:13:59 STEP 1998 H=1998:00 +CPU=116.123
  14:15:20 STEP 1999 H=1999:00 +CPU= 70.393
  14:16:46 STEP 2000 H=2000:00 +CPU= 72.732
  14:18:59 STEP 2001 H=2001:00 +CPU=113.454
  14:20:24 STEP 2002 H=2002:00 +CPU= 72.816
  14:21:44 STEP 2003 H=2003:00 +CPU= 69.892
  14:24:11 STEP 2004 H=2004:00 +CPU=127.258
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002004
  14:25:34 STEP 2005 H=2005:00 +CPU= 70.041
  14:26:57 STEP 2006 H=2006:00 +CPU= 72.500
  14:29:06 STEP 2007 H=2007:00 +CPU=113.217
  14:30:29 STEP 2008 H=2008:00 +CPU= 72.202
  14:31:46 STEP 2009 H=2009:00 +CPU= 70.018
  14:34:00 STEP 2010 H=2010:00 +CPU=115.881
  14:35:22 STEP 2011 H=2011:00 +CPU= 70.545
  14:36:45 STEP 2012 H=2012:00 +CPU= 72.535
  14:38:58 STEP 2013 H=2013:00 +CPU=112.742
  14:40:24 STEP 2014 H=2014:00 +CPU= 72.841
  14:41:49 STEP 2015 H=2015:00 +CPU= 72.760
  14:44:19 STEP 2016 H=2016:00 +CPU=132.684
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_83.zip
Uploading the intermediate file: upload_file_83.zip
Uploading trickle at timestep: 7257600
  14:45:39 STEP 2017 H=2017:00 +CPU= 69.502
  14:47:05 STEP 2018 H=2018:00 +CPU= 72.804
  14:49:14 STEP 2019 H=2019:00 +CPU=113.658
  14:50:39 STEP 2020 H=2020:00 +CPU= 73.167
  14:52:01 STEP 2021 H=2021:00 +CPU= 70.002
  14:54:15 STEP 2022 H=2022:00 +CPU=116.232
  14:55:37 STEP 2023 H=2023:00 +CPU= 70.078
  14:57:01 STEP 2024 H=2024:00 +CPU= 72.553
  14:59:12 STEP 2025 H=2025:00 +CPU=113.174
  15:00:38 STEP 2026 H=2026:00 +CPU= 72.902
  15:02:00 STEP 2027 H=2027:00 +CPU= 70.188
  15:04:25 STEP 2028 H=2028:00 +CPU=126.592
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002028
  15:05:48 STEP 2029 H=2029:00 +CPU= 70.066
  15:07:12 STEP 2030 H=2030:00 +CPU= 72.462
  15:09:20 STEP 2031 H=2031:00 +CPU=113.831
  15:10:45 STEP 2032 H=2032:00 +CPU= 72.499
  15:12:06 STEP 2033 H=2033:00 +CPU= 69.726
  15:14:21 STEP 2034 H=2034:00 +CPU=116.399
  15:15:45 STEP 2035 H=2035:00 +CPU= 69.789
  15:17:06 STEP 2036 H=2036:00 +CPU= 72.073
  15:19:15 STEP 2037 H=2037:00 +CPU=113.494
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1998050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0298
(argv4) batchid: 967
(argv5) wuid: 12183942
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_440bb838da3baa06ee7f983e27a4a748 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0298_1998050100_123_967_12183942.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0298_1998050100_123_967_12183942.zip
ic_ancil_file: ic_ancil_12183942
ifsdata_file: ifsdata_12183942
climate_data_file: clim_data_12183942
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ffe74f52377da88b52b146d1108344df to: /var/lib/boinc-client/slots/2/ic_ancil_12183942.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12183942.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a1c107519f828b7c80dd3a860f55b66b to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12183942.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12183942.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12183942.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12183942.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12183942.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12183942.xml
-- Model is restarting --
Adjusting last_iter, 2038, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12183942.xml
last_cpu_time: 186482
upload_file_number: 84
last_iter: 2015
last_upload: 7257600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159
The child process has been launched with process id: 3378016
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3378016:3378016] [20230129:205345:1675025625:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:56:21 STEP 2016 H=2016:00 +CPU=128.927
  20:57:41 STEP 2017 H=2017:00 +CPU= 69.070
  20:59:07 STEP 2018 H=2018:00 +CPU= 72.248
  21:01:15 STEP 2019 H=2019:00 +CPU=112.591
  21:02:38 STEP 2020 H=2020:00 +CPU= 72.056
  21:03:58 STEP 2021 H=2021:00 +CPU= 70.334
  21:06:15 STEP 2022 H=2022:00 +CPU=117.405
  21:07:36 STEP 2023 H=2023:00 +CPU= 69.433
  21:08:59 STEP 2024 H=2024:00 +CPU= 73.115
  21:11:10 STEP 2025 H=2025:00 +CPU=113.040
  21:12:36 STEP 2026 H=2026:00 +CPU= 72.187
  21:13:58 STEP 2027 H=2027:00 +CPU= 70.108
  21:16:29 STEP 2028 H=2028:00 +CPU=126.267
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002028
  21:17:50 STEP 2029 H=2029:00 +CPU= 69.604
  21:19:16 STEP 2030 H=2030:00 +CPU= 72.677
  21:21:28 STEP 2031 H=2031:00 +CPU=113.123
  21:22:51 STEP 2032 H=2032:00 +CPU= 72.223
  21:24:14 STEP 2033 H=2033:00 +CPU= 69.813
  21:26:29 STEP 2034 H=2034:00 +CPU=116.146
  21:27:50 STEP 2035 H=2035:00 +CPU= 69.718
  21:29:15 STEP 2036 H=2036:00 +CPU= 72.567
  21:31:26 STEP 2037 H=2037:00 +CPU=113.123
  21:32:47 STEP 2038 H=2038:00 +CPU= 71.884
  21:34:11 STEP 2039 H=2039:00 +CPU= 72.445
  21:36:47 STEP 2040 H=2040:00 +CPU=132.874
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_84.zip
Uploading the intermediate file: upload_file_84.zip
  21:38:09 STEP 2041 H=2041:00 +CPU= 69.581
  21:39:33 STEP 2042 H=2042:00 +CPU= 72.768
  21:41:47 STEP 2043 H=2043:00 +CPU=114.340
  21:43:09 STEP 2044 H=2044:00 +CPU= 72.474
  21:44:29 STEP 2045 H=2045:00 +CPU= 69.861
  21:46:45 STEP 2046 H=2046:00 +CPU=116.266
  21:48:07 STEP 2047 H=2047:00 +CPU= 69.790
  21:49:30 STEP 2048 H=2048:00 +CPU= 72.763
  21:51:40 STEP 2049 H=2049:00 +CPU=112.767
  21:53:04 STEP 2050 H=2050:00 +CPU= 72.841
  21:54:21 STEP 2051 H=2051:00 +CPU= 69.127
  21:56:48 STEP 2052 H=2052:00 +CPU=127.853
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002052
  21:58:09 STEP 2053 H=2053:00 +CPU= 69.536
  21:59:34 STEP 2054 H=2054:00 +CPU= 72.596
  22:01:48 STEP 2055 H=2055:00 +CPU=113.542
  22:03:13 STEP 2056 H=2056:00 +CPU= 72.400
  22:04:35 STEP 2057 H=2057:00 +CPU= 69.481
  22:06:52 STEP 2058 H=2058:00 +CPU=116.267
  22:08:15 STEP 2059 H=2059:00 +CPU= 69.956
  22:09:31 STEP 2060 H=2060:00 +CPU= 72.216
  22:11:41 STEP 2061 H=2061:00 +CPU=113.567
  22:13:02 STEP 2062 H=2062:00 +CPU= 72.516
  22:14:27 STEP 2063 H=2063:00 +CPU= 72.304
  22:17:03 STEP 2064 H=2064:00 +CPU=132.136
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_85.zip
Uploading the intermediate file: upload_file_85.zip
  22:18:26 STEP 2065 H=2065:00 +CPU= 69.353
  22:19:51 STEP 2066 H=2066:00 +CPU= 72.765
  22:22:02 STEP 2067 H=2067:00 +CPU=113.431
  22:23:27 STEP 2068 H=2068:00 +CPU= 72.209
  22:24:49 STEP 2069 H=2069:00 +CPU= 69.721
  22:27:06 STEP 2070 H=2070:00 +CPU=117.142
  22:28:25 STEP 2071 H=2071:00 +CPU= 68.976
  22:29:49 STEP 2072 H=2072:00 +CPU= 72.806
  22:32:01 STEP 2073 H=2073:00 +CPU=113.295
  22:33:25 STEP 2074 H=2074:00 +CPU= 72.248
  22:34:47 STEP 2075 H=2075:00 +CPU= 69.956
  22:37:17 STEP 2076 H=2076:00 +CPU=126.423
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002076
  22:38:36 STEP 2077 H=2077:00 +CPU= 69.463
  22:40:00 STEP 2078 H=2078:00 +CPU= 72.689
  22:42:09 STEP 2079 H=2079:00 +CPU=112.477
  22:43:33 STEP 2080 H=2080:00 +CPU= 72.657
  22:44:53 STEP 2081 H=2081:00 +CPU= 69.606
  22:47:07 STEP 2082 H=2082:00 +CPU=116.404
  22:48:30 STEP 2083 H=2083:00 +CPU= 70.047
  22:49:53 STEP 2084 H=2084:00 +CPU= 72.253
  22:52:05 STEP 2085 H=2085:00 +CPU=113.140
  22:53:30 STEP 2086 H=2086:00 +CPU= 72.519
  22:54:52 STEP 2087 H=2087:00 +CPU= 72.012
  22:57:27 STEP 2088 H=2088:00 +CPU=132.142
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_86.zip
Uploading the intermediate file: upload_file_86.zip
  22:58:49 STEP 2089 H=2089:00 +CPU= 69.961
  23:00:13 STEP 2090 H=2090:00 +CPU= 72.132
  23:02:21 STEP 2091 H=2091:00 +CPU=113.349
  23:03:46 STEP 2092 H=2092:00 +CPU= 72.267
  23:05:09 STEP 2093 H=2093:00 +CPU= 69.726
  23:07:24 STEP 2094 H=2094:00 +CPU=116.395
  23:08:46 STEP 2095 H=2095:00 +CPU= 69.807
  23:10:09 STEP 2096 H=2096:00 +CPU= 72.176
  23:12:18 STEP 2097 H=2097:00 +CPU=113.119
  23:13:40 STEP 2098 H=2098:00 +CPU= 71.998
  23:14:56 STEP 2099 H=2099:00 +CPU= 69.844
  23:17:22 STEP 2100 H=2100:00 +CPU=126.203
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002100
  23:18:44 STEP 2101 H=2101:00 +CPU= 69.957
  23:20:08 STEP 2102 H=2102:00 +CPU= 72.647
  23:22:17 STEP 2103 H=2103:00 +CPU=113.294
  23:23:41 STEP 2104 H=2104:00 +CPU= 72.068
  23:25:03 STEP 2105 H=2105:00 +CPU= 69.958
  23:27:18 STEP 2106 H=2106:00 +CPU=116.905
  23:28:38 STEP 2107 H=2107:00 +CPU= 69.359
  23:30:03 STEP 2108 H=2108:00 +CPU= 72.668
  23:32:16 STEP 2109 H=2109:00 +CPU=113.597
  23:33:40 STEP 2110 H=2110:00 +CPU= 72.516
  23:35:08 STEP 2111 H=2111:00 +CPU= 72.687
  23:37:44 STEP 2112 H=2112:00 +CPU=133.038
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_87.zip
Uploading the intermediate file: upload_file_87.zip
  23:39:05 STEP 2113 H=2113:00 +CPU= 70.096
  23:40:31 STEP 2114 H=2114:00 +CPU= 72.772
  23:42:42 STEP 2115 H=2115:00 +CPU=113.819
  23:44:05 STEP 2116 H=2116:00 +CPU= 72.801
  23:45:31 STEP 2117 H=2117:00 +CPU= 71.730
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:48:19 STEP 2118 H=2118:00 +CPU=131.453
  23:49:56 STEP 2119 H=2119:00 +CPU= 78.310
  23:51:35 STEP 2120 H=2120:00 +CPU= 81.565
  23:53:58 STEP 2121 H=2121:00 +CPU=120.471
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:56:52 STEP 2122 H=2122:00 +CPU= 73.402
  23:58:18 STEP 2123 H=2123:00 +CPU= 70.419
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:02:59 STEP 2124 H=2124:00 +CPU=127.615
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002124
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:06:27 STEP 2125 H=2125:00 +CPU= 70.126
  00:07:51 STEP 2126 H=2126:00 +CPU= 72.944
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:12:19 STEP 2127 H=2127:00 +CPU=113.827
  00:13:55 STEP 2128 H=2128:00 +CPU= 78.642
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:15:57 STEP 2129 H=2129:00 +CPU= 77.960
  00:18:36 STEP 2130 H=2130:00 +CPU=131.667
  00:20:10 STEP 2131 H=2131:00 +CPU= 78.049
  00:21:36 STEP 2132 H=2132:00 +CPU= 74.415
  00:23:49 STEP 2133 H=2133:00 +CPU=114.025
  00:25:13 STEP 2134 H=2134:00 +CPU= 72.884
  00:26:37 STEP 2135 H=2135:00 +CPU= 72.776
  00:29:13 STEP 2136 H=2136:00 +CPU=133.304
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_88.zip
Uploading the intermediate file: upload_file_88.zip
  00:30:37 STEP 2137 H=2137:00 +CPU= 70.104
  00:32:06 STEP 2138 H=2138:00 +CPU= 73.072
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:34:44 STEP 2139 H=2139:00 +CPU=122.772
  00:36:27 STEP 2140 H=2140:00 +CPU= 81.347
  00:38:03 STEP 2141 H=2141:00 +CPU= 78.205
  00:40:35 STEP 2142 H=2142:00 +CPU=129.480
  00:42:00 STEP 2143 H=2143:00 +CPU= 70.147
  00:43:23 STEP 2144 H=2144:00 +CPU= 72.673
  00:45:30 STEP 2145 H=2145:00 +CPU=113.824
  00:46:56 STEP 2146 H=2146:00 +CPU= 73.059
  00:48:18 STEP 2147 H=2147:00 +CPU= 70.176
  00:50:50 STEP 2148 H=2148:00 +CPU=127.285
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002148
  00:52:11 STEP 2149 H=2149:00 +CPU= 70.134
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:53:58 STEP 2150 H=2150:00 +CPU= 79.204
  00:56:33 STEP 2151 H=2151:00 +CPU=128.039
  00:58:12 STEP 2152 H=2152:00 +CPU= 81.715
  00:59:49 STEP 2153 H=2153:00 +CPU= 78.566
  01:02:06 STEP 2154 H=2154:00 +CPU=117.934
  01:03:26 STEP 2155 H=2155:00 +CPU= 70.225
  01:04:52 STEP 2156 H=2156:00 +CPU= 73.055
  01:07:07 STEP 2157 H=2157:00 +CPU=113.933
  01:08:31 STEP 2158 H=2158:00 +CPU= 72.892
  01:09:53 STEP 2159 H=2159:00 +CPU= 72.834
  01:12:33 STEP 2160 H=2160:00 +CPU=134.701
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_89.zip
Uploading the intermediate file: upload_file_89.zip
  01:14:10 STEP 2161 H=2161:00 +CPU= 78.352
  01:15:51 STEP 2162 H=2162:00 +CPU= 81.182
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:18:33 STEP 2163 H=2163:00 +CPU=127.596
  01:20:06 STEP 2164 H=2164:00 +CPU= 78.647
  01:21:27 STEP 2165 H=2165:00 +CPU= 69.894
  01:23:42 STEP 2166 H=2166:00 +CPU=116.965
  01:25:06 STEP 2167 H=2167:00 +CPU= 70.264
  01:26:28 STEP 2168 H=2168:00 +CPU= 72.819
  01:28:38 STEP 2169 H=2169:00 +CPU=114.050
  01:30:03 STEP 2170 H=2170:00 +CPU= 73.061
  01:31:23 STEP 2171 H=2171:00 +CPU= 70.147
  01:34:07 STEP 2172 H=2172:00 +CPU=137.484
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002172
  01:35:45 STEP 2173 H=2173:00 +CPU= 78.459
  01:37:27 STEP 2174 H=2174:00 +CPU= 81.827
  01:39:56 STEP 2175 H=2175:00 +CPU=124.225
  01:41:22 STEP 2176 H=2176:00 +CPU= 73.024
  01:42:46 STEP 2177 H=2177:00 +CPU= 70.087
  01:45:03 STEP 2178 H=2178:00 +CPU=117.150
  01:46:24 STEP 2179 H=2179:00 +CPU= 70.279
  01:47:50 STEP 2180 H=2180:00 +CPU= 72.980
  01:50:04 STEP 2181 H=2181:00 +CPU=113.931
  01:51:31 STEP 2182 H=2182:00 +CPU= 72.765
  01:53:05 STEP 2183 H=2183:00 +CPU= 80.866
  01:56:05 STEP 2184 H=2184:00 +CPU=149.976
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_90.zip
Uploading the intermediate file: upload_file_90.zip
  01:57:39 STEP 2185 H=2185:00 +CPU= 78.221
  01:59:10 STEP 2186 H=2186:00 +CPU= 78.164
  02:01:23 STEP 2187 H=2187:00 +CPU=113.765
  02:02:47 STEP 2188 H=2188:00 +CPU= 72.925
  02:04:08 STEP 2189 H=2189:00 +CPU= 70.020
  02:06:26 STEP 2190 H=2190:00 +CPU=117.474
  02:07:48 STEP 2191 H=2191:00 +CPU= 70.263
  02:09:12 STEP 2192 H=2192:00 +CPU= 73.018
  02:11:29 STEP 2193 H=2193:00 +CPU=117.586
  02:13:05 STEP 2194 H=2194:00 +CPU= 81.478
  02:14:39 STEP 2195 H=2195:00 +CPU= 78.629
  02:17:08 STEP 2196 H=2196:00 +CPU=128.090
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002196
  02:18:31 STEP 2197 H=2197:00 +CPU= 70.325
  02:19:57 STEP 2198 H=2198:00 +CPU= 73.236
  02:22:10 STEP 2199 H=2199:00 +CPU=114.289
  02:23:37 STEP 2200 H=2200:00 +CPU= 73.020
  02:25:01 STEP 2201 H=2201:00 +CPU= 70.191
  02:27:23 STEP 2202 H=2202:00 +CPU=119.899
  02:28:53 STEP 2203 H=2203:00 +CPU= 78.081
  02:30:34 STEP 2204 H=2204:00 +CPU= 81.453
  02:32:44 STEP 2205 H=2205:00 +CPU=114.493
  02:34:07 STEP 2206 H=2206:00 +CPU= 72.648
  02:35:31 STEP 2207 H=2207:00 +CPU= 72.777
  02:38:08 STEP 2208 H=2208:00 +CPU=133.440
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_91.zip
Uploading the intermediate file: upload_file_91.zip
  02:39:30 STEP 2209 H=2209:00 +CPU= 70.309
  02:40:55 STEP 2210 H=2210:00 +CPU= 73.013
  02:43:07 STEP 2211 H=2211:00 +CPU=115.328
  02:44:48 STEP 2212 H=2212:00 +CPU= 81.729
  02:46:22 STEP 2213 H=2213:00 +CPU= 78.846
  02:49:04 STEP 2214 H=2214:00 +CPU=132.639
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:50:47 STEP 2215 H=2215:00 +CPU= 75.534
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1998050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0298
(argv4) batchid: 967
(argv5) wuid: 12183942
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_440bb838da3baa06ee7f983e27a4a748 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0298_1998050100_123_967_12183942.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0298_1998050100_123_967_12183942.zip
ic_ancil_file: ic_ancil_12183942
ifsdata_file: ifsdata_12183942
climate_data_file: clim_data_12183942
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ffe74f52377da88b52b146d1108344df to: /var/lib/boinc-client/slots/2/ic_ancil_12183942.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12183942.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a1c107519f828b7c80dd3a860f55b66b to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12183942.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12183942.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12183942.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12183942.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12183942.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12183942.xml
-- Model is restarting --
Adjusting last_iter, 2216, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12183942.xml
last_cpu_time: 204286
upload_file_number: 92
last_iter: 2207
last_upload: 7948800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159
The child process has been launched with process id: 3436465
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3436465:3436465] [20230130:161754:1675095474:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:20:33 STEP 2208 H=2208:00 +CPU=127.475
  16:21:55 STEP 2209 H=2209:00 +CPU= 68.524
  16:23:17 STEP 2210 H=2210:00 +CPU= 71.741
  16:25:27 STEP 2211 H=2211:00 +CPU=111.932
  16:26:49 STEP 2212 H=2212:00 +CPU= 71.626
  16:28:09 STEP 2213 H=2213:00 +CPU= 68.939
  16:30:23 STEP 2214 H=2214:00 +CPU=114.991
  16:31:41 STEP 2215 H=2215:00 +CPU= 68.998
  16:33:04 STEP 2216 H=2216:00 +CPU= 71.671
  16:35:14 STEP 2217 H=2217:00 +CPU=111.942
  16:36:36 STEP 2218 H=2218:00 +CPU= 71.227
  16:37:53 STEP 2219 H=2219:00 +CPU= 68.626
  16:40:18 STEP 2220 H=2220:00 +CPU=124.954
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002220
  16:41:37 STEP 2221 H=2221:00 +CPU= 68.763
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:43:31 STEP 2222 H=2222:00 +CPU= 71.956
  16:45:38 STEP 2223 H=2223:00 +CPU=111.605
  16:47:01 STEP 2224 H=2224:00 +CPU= 71.608
  16:48:21 STEP 2225 H=2225:00 +CPU= 69.172
  16:50:34 STEP 2226 H=2226:00 +CPU=115.219
  16:51:53 STEP 2227 H=2227:00 +CPU= 68.878
  16:53:18 STEP 2228 H=2228:00 +CPU= 71.721
  16:55:29 STEP 2229 H=2229:00 +CPU=112.389
  16:56:53 STEP 2230 H=2230:00 +CPU= 71.954
  16:58:17 STEP 2231 H=2231:00 +CPU= 71.854
  17:00:49 STEP 2232 H=2232:00 +CPU=130.948
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183942/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_1998050100_123_967_12183942_2_r168525159_92.zip
Uploading the intermediate file: upload_file_92.zip
  17:02:11 STEP 2233 H=2233:00 +CPU= 69.005
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:44:59 1511650 22304010 oifs_43r3_ps_0298_1998050100_123_967_12183942_2 7,257,600 184,616 0.0254
09 May 2023 16:44:41 1511650 22304010 oifs_43r3_ps_0298_1998050100_123_967_12183942_2 5,702,400 146,050 0.0256
09 May 2023 16:43:55 1511650 22304010 oifs_43r3_ps_0298_1998050100_123_967_12183942_2 3,628,800 92,094 0.0254
09 May 2023 16:43:39 1511650 22304010 oifs_43r3_ps_0298_1998050100_123_967_12183942_2 2,764,800 71,615 0.0259
09 May 2023 16:43:09 1511650 22304010 oifs_43r3_ps_0298_1998050100_123_967_12183942_2 1,728,000 43,633 0.0253
09 May 2023 16:42:45 1511650 22304010 oifs_43r3_ps_0298_1998050100_123_967_12183942_2 864,000 21,634 0.0250


©2024 cpdn.org