climateprediction.net home page
Task 22251773

Task 22251773

Name oifs_43r3_ps_0904_1981050100_123_950_12167548_0
Workunit 12167548
Created 20 Dec 2022, 20:44:21 UTC
Sent 21 Jan 2023, 9:10:29 UTC
Report deadline 22 Mar 2023, 9:10:29 UTC
Received 22 Jan 2023, 12:33:38 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1531398
Run time 13 hours 29 min 20 sec
CPU time 13 hours 10 min 57 sec
Validate state Invalid
Credit 10,160.64
Device peak FLOPS 7.92 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,804.20 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,917.87 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
6 H=1866:00 +CPU= 24.508
  09:11:19 STEP 1867 H=1867:00 +CPU= 16.594
  09:11:36 STEP 1868 H=1868:00 +CPU= 16.956
  09:12:01 STEP 1869 H=1869:00 +CPU= 25.339
  09:12:17 STEP 1870 H=1870:00 +CPU= 16.045
  09:12:34 STEP 1871 H=1871:00 +CPU= 16.243
  09:13:03 STEP 1872 H=1872:00 +CPU= 29.077
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_77.zip
Uploading the intermediate file: upload_file_77.zip
  09:13:18 STEP 1873 H=1873:00 +CPU= 15.504
  09:13:34 STEP 1874 H=1874:00 +CPU= 15.907
  09:13:59 STEP 1875 H=1875:00 +CPU= 24.580
  09:14:16 STEP 1876 H=1876:00 +CPU= 17.023
  09:14:32 STEP 1877 H=1877:00 +CPU= 15.550
  09:14:57 STEP 1878 H=1878:00 +CPU= 25.709
  09:15:14 STEP 1879 H=1879:00 +CPU= 16.651
  09:15:30 STEP 1880 H=1880:00 +CPU= 16.517
  09:15:56 STEP 1881 H=1881:00 +CPU= 25.297
  09:16:13 STEP 1882 H=1882:00 +CPU= 16.758
  09:16:28 STEP 1883 H=1883:00 +CPU= 15.398
  09:16:55 STEP 1884 H=1884:00 +CPU= 26.662
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+001884
  09:17:10 STEP 1885 H=1885:00 +CPU= 15.557
  09:17:26 STEP 1886 H=1886:00 +CPU= 15.738
  09:17:50 STEP 1887 H=1887:00 +CPU= 24.243
  09:18:07 STEP 1888 H=1888:00 +CPU= 16.261
  09:18:22 STEP 1889 H=1889:00 +CPU= 15.480
  09:18:47 STEP 1890 H=1890:00 +CPU= 25.025
  09:19:03 STEP 1891 H=1891:00 +CPU= 15.438
  09:19:19 STEP 1892 H=1892:00 +CPU= 16.035
  09:19:44 STEP 1893 H=1893:00 +CPU= 24.706
  09:20:00 STEP 1894 H=1894:00 +CPU= 16.068
  09:20:16 STEP 1895 H=1895:00 +CPU= 16.005
  09:20:46 STEP 1896 H=1896:00 +CPU= 28.971
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_78.zip
Uploading the intermediate file: upload_file_78.zip
  09:21:02 STEP 1897 H=1897:00 +CPU= 15.935
  09:21:18 STEP 1898 H=1898:00 +CPU= 15.797
  09:21:42 STEP 1899 H=1899:00 +CPU= 23.996
  09:21:59 STEP 1900 H=1900:00 +CPU= 16.746
  09:22:15 STEP 1901 H=1901:00 +CPU= 15.422
  09:22:39 STEP 1902 H=1902:00 +CPU= 24.513
  09:22:55 STEP 1903 H=1903:00 +CPU= 15.739
  09:23:11 STEP 1904 H=1904:00 +CPU= 16.235
  09:23:35 STEP 1905 H=1905:00 +CPU= 23.762
  09:23:51 STEP 1906 H=1906:00 +CPU= 16.133
  09:24:07 STEP 1907 H=1907:00 +CPU= 16.371
  09:24:35 STEP 1908 H=1908:00 +CPU= 27.502
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+001908
  09:24:51 STEP 1909 H=1909:00 +CPU= 15.661
  09:25:07 STEP 1910 H=1910:00 +CPU= 16.146
  09:25:31 STEP 1911 H=1911:00 +CPU= 24.299
  09:25:47 STEP 1912 H=1912:00 +CPU= 16.256
  09:26:04 STEP 1913 H=1913:00 +CPU= 16.407
  09:26:29 STEP 1914 H=1914:00 +CPU= 25.298
  09:26:45 STEP 1915 H=1915:00 +CPU= 15.538
  09:27:01 STEP 1916 H=1916:00 +CPU= 16.434
  09:27:27 STEP 1917 H=1917:00 +CPU= 24.894
  09:27:43 STEP 1918 H=1918:00 +CPU= 16.496
  09:27:59 STEP 1919 H=1919:00 +CPU= 15.858
  09:28:28 STEP 1920 H=1920:00 +CPU= 28.976
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_79.zip
Uploading the intermediate file: upload_file_79.zip
  09:28:44 STEP 1921 H=1921:00 +CPU= 15.635
  09:28:59 STEP 1922 H=1922:00 +CPU= 15.678
  09:29:24 STEP 1923 H=1923:00 +CPU= 24.109
  09:29:40 STEP 1924 H=1924:00 +CPU= 16.015
  09:29:55 STEP 1925 H=1925:00 +CPU= 15.405
  09:30:20 STEP 1926 H=1926:00 +CPU= 24.843
  09:30:35 STEP 1927 H=1927:00 +CPU= 15.496
  09:30:51 STEP 1928 H=1928:00 +CPU= 16.020
  09:31:15 STEP 1929 H=1929:00 +CPU= 23.914
  09:31:32 STEP 1930 H=1930:00 +CPU= 16.455
  09:31:48 STEP 1931 H=1931:00 +CPU= 15.748
  09:32:15 STEP 1932 H=1932:00 +CPU= 27.177
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+001932
  09:32:30 STEP 1933 H=1933:00 +CPU= 15.463
  09:32:46 STEP 1934 H=1934:00 +CPU= 15.949
  09:33:12 STEP 1935 H=1935:00 +CPU= 25.090
  09:33:28 STEP 1936 H=1936:00 +CPU= 16.197
  09:33:43 STEP 1937 H=1937:00 +CPU= 15.429
  09:34:08 STEP 1938 H=1938:00 +CPU= 24.494
  09:34:23 STEP 1939 H=1939:00 +CPU= 15.387
  09:34:39 STEP 1940 H=1940:00 +CPU= 16.213
  09:35:04 STEP 1941 H=1941:00 +CPU= 24.480
  09:35:21 STEP 1942 H=1942:00 +CPU= 16.846
  09:35:38 STEP 1943 H=1943:00 +CPU= 16.215
  09:36:07 STEP 1944 H=1944:00 +CPU= 29.266
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_80.zip
Uploading the intermediate file: upload_file_80.zip
  09:36:23 STEP 1945 H=1945:00 +CPU= 15.746
  09:36:39 STEP 1946 H=1946:00 +CPU= 16.240
  09:37:03 STEP 1947 H=1947:00 +CPU= 23.984
  09:37:19 STEP 1948 H=1948:00 +CPU= 16.058
  09:37:34 STEP 1949 H=1949:00 +CPU= 15.229
  09:38:00 STEP 1950 H=1950:00 +CPU= 25.667
  09:38:16 STEP 1951 H=1951:00 +CPU= 15.408
  09:38:32 STEP 1952 H=1952:00 +CPU= 15.930
  09:38:56 STEP 1953 H=1953:00 +CPU= 24.454
  09:39:12 STEP 1954 H=1954:00 +CPU= 16.075
  09:39:28 STEP 1955 H=1955:00 +CPU= 15.257
  09:39:55 STEP 1956 H=1956:00 +CPU= 27.810
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+001956
  09:40:12 STEP 1957 H=1957:00 +CPU= 16.268
  09:40:28 STEP 1958 H=1958:00 +CPU= 16.302
  09:40:53 STEP 1959 H=1959:00 +CPU= 24.553
  09:41:10 STEP 1960 H=1960:00 +CPU= 16.773
  09:41:25 STEP 1961 H=1961:00 +CPU= 15.803
  09:41:50 STEP 1962 H=1962:00 +CPU= 24.363
  09:42:05 STEP 1963 H=1963:00 +CPU= 15.337
  09:42:21 STEP 1964 H=1964:00 +CPU= 15.874
  09:42:46 STEP 1965 H=1965:00 +CPU= 24.611
  09:43:03 STEP 1966 H=1966:00 +CPU= 16.779
  09:43:19 STEP 1967 H=1967:00 +CPU= 16.602
  09:43:48 STEP 1968 H=1968:00 +CPU= 28.643
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_81.zip
Uploading the intermediate file: upload_file_81.zip
  09:44:04 STEP 1969 H=1969:00 +CPU= 16.348
  09:44:21 STEP 1970 H=1970:00 +CPU= 16.660
  09:44:46 STEP 1971 H=1971:00 +CPU= 25.312
  09:45:03 STEP 1972 H=1972:00 +CPU= 16.696
  09:45:19 STEP 1973 H=1973:00 +CPU= 16.401
  09:45:45 STEP 1974 H=1974:00 +CPU= 25.440
  09:46:00 STEP 1975 H=1975:00 +CPU= 15.537
  09:46:17 STEP 1976 H=1976:00 +CPU= 16.366
  09:46:41 STEP 1977 H=1977:00 +CPU= 24.200
  09:46:57 STEP 1978 H=1978:00 +CPU= 16.082
  09:47:13 STEP 1979 H=1979:00 +CPU= 15.391
  09:47:39 STEP 1980 H=1980:00 +CPU= 26.641
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+001980
  09:47:55 STEP 1981 H=1981:00 +CPU= 15.719
  09:48:12 STEP 1982 H=1982:00 +CPU= 16.458
  09:48:36 STEP 1983 H=1983:00 +CPU= 24.305
  09:48:52 STEP 1984 H=1984:00 +CPU= 16.326
  09:49:08 STEP 1985 H=1985:00 +CPU= 15.789
  09:49:34 STEP 1986 H=1986:00 +CPU= 25.549
  09:49:50 STEP 1987 H=1987:00 +CPU= 16.526
  09:50:07 STEP 1988 H=1988:00 +CPU= 16.387
  09:50:31 STEP 1989 H=1989:00 +CPU= 24.723
  09:50:48 STEP 1990 H=1990:00 +CPU= 16.868
  09:51:05 STEP 1991 H=1991:00 +CPU= 16.394
  09:51:34 STEP 1992 H=1992:00 +CPU= 28.662
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_82.zip
Uploading the intermediate file: upload_file_82.zip
  09:51:49 STEP 1993 H=1993:00 +CPU= 15.722
  09:52:05 STEP 1994 H=1994:00 +CPU= 15.916
  09:52:29 STEP 1995 H=1995:00 +CPU= 23.863
  09:52:45 STEP 1996 H=1996:00 +CPU= 16.147
  09:53:01 STEP 1997 H=1997:00 +CPU= 15.386
  09:53:25 STEP 1998 H=1998:00 +CPU= 24.385
  09:53:41 STEP 1999 H=1999:00 +CPU= 15.746
  09:53:57 STEP 2000 H=2000:00 +CPU= 15.831
  09:54:21 STEP 2001 H=2001:00 +CPU= 24.337
  09:54:37 STEP 2002 H=2002:00 +CPU= 16.251
  09:54:53 STEP 2003 H=2003:00 +CPU= 15.602
  09:55:20 STEP 2004 H=2004:00 +CPU= 27.056
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002004
  09:55:36 STEP 2005 H=2005:00 +CPU= 15.660
  09:55:52 STEP 2006 H=2006:00 +CPU= 15.989
  09:56:17 STEP 2007 H=2007:00 +CPU= 25.024
  09:56:33 STEP 2008 H=2008:00 +CPU= 16.428
  09:56:50 STEP 2009 H=2009:00 +CPU= 16.120
  09:57:15 STEP 2010 H=2010:00 +CPU= 25.711
  09:57:31 STEP 2011 H=2011:00 +CPU= 15.951
  09:57:48 STEP 2012 H=2012:00 +CPU= 16.362
  09:58:13 STEP 2013 H=2013:00 +CPU= 24.479
  09:58:29 STEP 2014 H=2014:00 +CPU= 16.199
  09:58:45 STEP 2015 H=2015:00 +CPU= 16.204
  09:59:15 STEP 2016 H=2016:00 +CPU= 29.138
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_83.zip
Uploading the intermediate file: upload_file_83.zip
  09:59:30 STEP 2017 H=2017:00 +CPU= 15.761
Uploading trickle at timestep: 7257600
  09:59:47 STEP 2018 H=2018:00 +CPU= 16.690
  10:00:13 STEP 2019 H=2019:00 +CPU= 25.408
  10:00:29 STEP 2020 H=2020:00 +CPU= 16.742
  10:00:45 STEP 2021 H=2021:00 +CPU= 16.086
  10:01:11 STEP 2022 H=2022:00 +CPU= 25.884
  10:01:27 STEP 2023 H=2023:00 +CPU= 15.562
  10:01:43 STEP 2024 H=2024:00 +CPU= 16.315
  10:02:09 STEP 2025 H=2025:00 +CPU= 25.162
  10:02:25 STEP 2026 H=2026:00 +CPU= 16.809
  10:02:41 STEP 2027 H=2027:00 +CPU= 15.985
  10:03:09 STEP 2028 H=2028:00 +CPU= 27.817
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002028
  10:03:25 STEP 2029 H=2029:00 +CPU= 15.491
  10:03:40 STEP 2030 H=2030:00 +CPU= 15.658
  10:04:06 STEP 2031 H=2031:00 +CPU= 25.190
  10:04:22 STEP 2032 H=2032:00 +CPU= 16.767
  10:04:38 STEP 2033 H=2033:00 +CPU= 15.455
  10:05:04 STEP 2034 H=2034:00 +CPU= 26.237
  10:05:21 STEP 2035 H=2035:00 +CPU= 16.341
  10:05:37 STEP 2036 H=2036:00 +CPU= 16.195
  10:06:02 STEP 2037 H=2037:00 +CPU= 25.128
  10:06:19 STEP 2038 H=2038:00 +CPU= 16.689
  10:06:34 STEP 2039 H=2039:00 +CPU= 15.341
  10:07:03 STEP 2040 H=2040:00 +CPU= 28.455
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_84.zip
Uploading the intermediate file: upload_file_84.zip
  10:07:19 STEP 2041 H=2041:00 +CPU= 16.027
  10:07:35 STEP 2042 H=2042:00 +CPU= 16.079
  10:08:01 STEP 2043 H=2043:00 +CPU= 25.519
  10:08:17 STEP 2044 H=2044:00 +CPU= 16.661
  10:08:32 STEP 2045 H=2045:00 +CPU= 15.133
  10:08:57 STEP 2046 H=2046:00 +CPU= 24.777
  10:09:13 STEP 2047 H=2047:00 +CPU= 15.619
  10:09:28 STEP 2048 H=2048:00 +CPU= 15.429
  10:09:53 STEP 2049 H=2049:00 +CPU= 24.373
  10:10:09 STEP 2050 H=2050:00 +CPU= 16.677
  10:10:25 STEP 2051 H=2051:00 +CPU= 15.134
  10:10:52 STEP 2052 H=2052:00 +CPU= 26.996
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002052
  10:11:07 STEP 2053 H=2053:00 +CPU= 15.570
  10:11:23 STEP 2054 H=2054:00 +CPU= 15.922
  10:11:48 STEP 2055 H=2055:00 +CPU= 24.179
  10:12:04 STEP 2056 H=2056:00 +CPU= 16.036
  10:12:19 STEP 2057 H=2057:00 +CPU= 15.403
  10:12:45 STEP 2058 H=2058:00 +CPU= 26.017
  10:13:01 STEP 2059 H=2059:00 +CPU= 15.563
  10:13:16 STEP 2060 H=2060:00 +CPU= 15.616
  10:13:41 STEP 2061 H=2061:00 +CPU= 24.358
  10:13:57 STEP 2062 H=2062:00 +CPU= 16.115
  10:14:13 STEP 2063 H=2063:00 +CPU= 16.104
  10:14:43 STEP 2064 H=2064:00 +CPU= 29.777
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_85.zip
Uploading the intermediate file: upload_file_85.zip
  10:15:00 STEP 2065 H=2065:00 +CPU= 16.450
  10:15:16 STEP 2066 H=2066:00 +CPU= 16.019
  10:15:41 STEP 2067 H=2067:00 +CPU= 25.377
  10:15:58 STEP 2068 H=2068:00 +CPU= 16.920
  10:16:14 STEP 2069 H=2069:00 +CPU= 15.444
  10:16:39 STEP 2070 H=2070:00 +CPU= 25.817
  10:16:56 STEP 2071 H=2071:00 +CPU= 16.383
  10:17:12 STEP 2072 H=2072:00 +CPU= 16.011
  10:17:36 STEP 2073 H=2073:00 +CPU= 24.045
  10:17:52 STEP 2074 H=2074:00 +CPU= 16.162
  10:18:08 STEP 2075 H=2075:00 +CPU= 15.250
  10:18:35 STEP 2076 H=2076:00 +CPU= 27.391
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002076
  10:18:51 STEP 2077 H=2077:00 +CPU= 16.283
  10:19:07 STEP 2078 H=2078:00 +CPU= 16.097
  10:19:32 STEP 2079 H=2079:00 +CPU= 24.003
  10:19:48 STEP 2080 H=2080:00 +CPU= 16.478
  10:20:04 STEP 2081 H=2081:00 +CPU= 15.957
  10:20:30 STEP 2082 H=2082:00 +CPU= 25.787
  10:20:46 STEP 2083 H=2083:00 +CPU= 16.431
  10:21:03 STEP 2084 H=2084:00 +CPU= 16.296
  10:21:27 STEP 2085 H=2085:00 +CPU= 24.621
  10:21:43 STEP 2086 H=2086:00 +CPU= 16.081
  10:22:00 STEP 2087 H=2087:00 +CPU= 16.079
  10:22:28 STEP 2088 H=2088:00 +CPU= 28.391
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_86.zip
Uploading the intermediate file: upload_file_86.zip
  10:22:44 STEP 2089 H=2089:00 +CPU= 16.195
  10:23:01 STEP 2090 H=2090:00 +CPU= 16.294
  10:23:26 STEP 2091 H=2091:00 +CPU= 24.879
  10:23:42 STEP 2092 H=2092:00 +CPU= 16.794
  10:23:58 STEP 2093 H=2093:00 +CPU= 15.627
  10:24:23 STEP 2094 H=2094:00 +CPU= 24.879
  10:24:39 STEP 2095 H=2095:00 +CPU= 15.587
  10:24:55 STEP 2096 H=2096:00 +CPU= 15.971
  10:25:20 STEP 2097 H=2097:00 +CPU= 24.826
  10:25:36 STEP 2098 H=2098:00 +CPU= 16.669
  10:25:52 STEP 2099 H=2099:00 +CPU= 15.404
  10:26:19 STEP 2100 H=2100:00 +CPU= 26.775
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002100
  10:26:34 STEP 2101 H=2101:00 +CPU= 15.679
  10:26:50 STEP 2102 H=2102:00 +CPU= 16.062
  10:27:15 STEP 2103 H=2103:00 +CPU= 24.211
  10:27:31 STEP 2104 H=2104:00 +CPU= 16.279
  10:27:47 STEP 2105 H=2105:00 +CPU= 15.502
  10:28:11 STEP 2106 H=2106:00 +CPU= 24.399
  10:28:27 STEP 2107 H=2107:00 +CPU= 16.188
  10:28:44 STEP 2108 H=2108:00 +CPU= 16.248
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0904
(argv4) batchid: 950
(argv5) wuid: 12167548
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/5
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6745e5301196e3837dcf642d186f007e to: /var/lib/boinc/slots/5/oifs_43r3_ps_0904_1981050100_123_950_12167548.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/5/oifs_43r3_ps_0904_1981050100_123_950_12167548.zip
ic_ancil_file: ic_ancil_12167548
ifsdata_file: ifsdata_12167548
climate_data_file: clim_data_12167548
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0aad1e958e4aec015bd3f8e586011e6c to: /var/lib/boinc/slots/5/ic_ancil_12167548.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/5/ic_ancil_12167548.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_67b1f4a8c0a9a550b3b76ef4a3ccfb2f to: /var/lib/boinc/slots/5/ifsdata/ifsdata_12167548.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/5/ifsdata/ifsdata_12167548.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/5/159l_2/clim_data_12167548.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/5/159l_2/clim_data_12167548.zip
Checking for progress XML file: /var/lib/boinc/slots/5/progress_file_12167548.xml
Opened progress file ok : /var/lib/boinc/slots/5/progress_file_12167548.xml
-- Model is restarting --
Adjusting last_iter, 2109, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/5/progress_file_12167548.xml
last_cpu_time: 43338
upload_file_number: 87
last_iter: 2087
last_upload: 7516800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060
The child process has been launched with process id: 31144
Executing the command: /var/lib/boinc/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:31144:31144] [20230122:103533:1674383733:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:35:57 STEP 2088 H=2088:00 +CPU= 22.281
  10:36:10 STEP 2089 H=2089:00 +CPU= 12.409
  10:36:23 STEP 2090 H=2090:00 +CPU= 12.920
  10:36:43 STEP 2091 H=2091:00 +CPU= 19.881
  10:36:56 STEP 2092 H=2092:00 +CPU= 12.938
  10:37:09 STEP 2093 H=2093:00 +CPU= 12.536
  10:37:30 STEP 2094 H=2094:00 +CPU= 20.476
  10:37:43 STEP 2095 H=2095:00 +CPU= 12.455
  10:37:56 STEP 2096 H=2096:00 +CPU= 12.952
  10:38:16 STEP 2097 H=2097:00 +CPU= 19.889
  10:38:29 STEP 2098 H=2098:00 +CPU= 12.955
  10:38:42 STEP 2099 H=2099:00 +CPU= 12.480
  10:39:04 STEP 2100 H=2100:00 +CPU= 22.355
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002100
  10:39:17 STEP 2101 H=2101:00 +CPU= 12.464
  10:39:30 STEP 2102 H=2102:00 +CPU= 12.907
  10:39:51 STEP 2103 H=2103:00 +CPU= 19.913
  10:40:05 STEP 2104 H=2104:00 +CPU= 13.006
  10:40:18 STEP 2105 H=2105:00 +CPU= 12.587
  10:40:39 STEP 2106 H=2106:00 +CPU= 20.402
  10:40:52 STEP 2107 H=2107:00 +CPU= 12.513
  10:41:05 STEP 2108 H=2108:00 +CPU= 12.918
  10:41:26 STEP 2109 H=2109:00 +CPU= 20.050
  10:41:40 STEP 2110 H=2110:00 +CPU= 13.049
  10:41:53 STEP 2111 H=2111:00 +CPU= 12.910
  10:42:17 STEP 2112 H=2112:00 +CPU= 23.573
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_87.zip
Uploading the intermediate file: upload_file_87.zip
  10:42:30 STEP 2113 H=2113:00 +CPU= 12.468
  10:42:44 STEP 2114 H=2114:00 +CPU= 13.038
  10:43:05 STEP 2115 H=2115:00 +CPU= 20.082
  10:43:18 STEP 2116 H=2116:00 +CPU= 13.047
  10:43:31 STEP 2117 H=2117:00 +CPU= 12.625
  10:43:52 STEP 2118 H=2118:00 +CPU= 20.466
  10:44:05 STEP 2119 H=2119:00 +CPU= 12.565
  10:44:19 STEP 2120 H=2120:00 +CPU= 13.039
  10:44:39 STEP 2121 H=2121:00 +CPU= 20.015
  10:44:53 STEP 2122 H=2122:00 +CPU= 12.994
  10:45:06 STEP 2123 H=2123:00 +CPU= 12.546
  10:45:29 STEP 2124 H=2124:00 +CPU= 22.448
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002124
  10:45:43 STEP 2125 H=2125:00 +CPU= 12.601
  10:45:56 STEP 2126 H=2126:00 +CPU= 13.080
  10:46:17 STEP 2127 H=2127:00 +CPU= 20.008
  10:46:30 STEP 2128 H=2128:00 +CPU= 12.980
  10:46:44 STEP 2129 H=2129:00 +CPU= 12.536
  10:47:04 STEP 2130 H=2130:00 +CPU= 20.346
  10:47:17 STEP 2131 H=2131:00 +CPU= 12.537
  10:47:31 STEP 2132 H=2132:00 +CPU= 13.018
  10:47:52 STEP 2133 H=2133:00 +CPU= 20.081
  10:48:05 STEP 2134 H=2134:00 +CPU= 12.845
  10:48:18 STEP 2135 H=2135:00 +CPU= 12.893
  10:48:42 STEP 2136 H=2136:00 +CPU= 23.463
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_88.zip
Uploading the intermediate file: upload_file_88.zip
  10:48:54 STEP 2137 H=2137:00 +CPU= 12.484
  10:49:08 STEP 2138 H=2138:00 +CPU= 13.000
  10:49:29 STEP 2139 H=2139:00 +CPU= 19.950
  10:49:42 STEP 2140 H=2140:00 +CPU= 12.958
  10:49:55 STEP 2141 H=2141:00 +CPU= 12.552
  10:50:16 STEP 2142 H=2142:00 +CPU= 20.451
  10:50:29 STEP 2143 H=2143:00 +CPU= 12.574
  10:50:43 STEP 2144 H=2144:00 +CPU= 13.002
  10:51:03 STEP 2145 H=2145:00 +CPU= 19.852
  10:51:17 STEP 2146 H=2146:00 +CPU= 13.093
  10:51:29 STEP 2147 H=2147:00 +CPU= 12.550
  10:51:52 STEP 2148 H=2148:00 +CPU= 22.312
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002148
  10:52:05 STEP 2149 H=2149:00 +CPU= 12.515
  10:52:18 STEP 2150 H=2150:00 +CPU= 12.965
  10:52:39 STEP 2151 H=2151:00 +CPU= 20.186
  10:52:52 STEP 2152 H=2152:00 +CPU= 12.870
  10:53:06 STEP 2153 H=2153:00 +CPU= 12.996
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:53:43 STEP 2154 H=2154:00 +CPU= 21.413
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:54:10 STEP 2155 H=2155:00 +CPU= 13.160
  10:54:24 STEP 2156 H=2156:00 +CPU= 13.563
  10:54:47 STEP 2157 H=2157:00 +CPU= 21.142
  10:55:01 STEP 2158 H=2158:00 +CPU= 13.410
  10:55:16 STEP 2159 H=2159:00 +CPU= 13.696
  10:55:46 STEP 2160 H=2160:00 +CPU= 25.257
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_89.zip
Uploading the intermediate file: upload_file_89.zip
  10:56:00 STEP 2161 H=2161:00 +CPU= 13.138
  10:56:14 STEP 2162 H=2162:00 +CPU= 13.564
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:57:36 STEP 2163 H=2163:00 +CPU= 21.023
  10:57:49 STEP 2164 H=2164:00 +CPU= 13.304
  10:58:03 STEP 2165 H=2165:00 +CPU= 13.213
  10:58:25 STEP 2166 H=2166:00 +CPU= 21.291
  10:58:38 STEP 2167 H=2167:00 +CPU= 12.793
  10:58:52 STEP 2168 H=2168:00 +CPU= 13.599
  10:59:13 STEP 2169 H=2169:00 +CPU= 20.844
  10:59:27 STEP 2170 H=2170:00 +CPU= 13.412
  10:59:41 STEP 2171 H=2171:00 +CPU= 13.122
  11:00:05 STEP 2172 H=2172:00 +CPU= 23.416
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002172
  11:00:18 STEP 2173 H=2173:00 +CPU= 13.012
  11:00:33 STEP 2174 H=2174:00 +CPU= 13.623
  11:00:54 STEP 2175 H=2175:00 +CPU= 20.614
  11:01:08 STEP 2176 H=2176:00 +CPU= 13.458
  11:01:21 STEP 2177 H=2177:00 +CPU= 13.024
  11:01:44 STEP 2178 H=2178:00 +CPU= 21.314
  11:01:57 STEP 2179 H=2179:00 +CPU= 12.838
  11:02:10 STEP 2180 H=2180:00 +CPU= 13.494
  11:02:32 STEP 2181 H=2181:00 +CPU= 20.736
  11:02:46 STEP 2182 H=2182:00 +CPU= 13.438
  11:03:00 STEP 2183 H=2183:00 +CPU= 13.544
  11:03:25 STEP 2184 H=2184:00 +CPU= 24.485
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_90.zip
Uploading the intermediate file: upload_file_90.zip
  11:03:39 STEP 2185 H=2185:00 +CPU= 13.008
  11:03:53 STEP 2186 H=2186:00 +CPU= 13.566
  11:04:15 STEP 2187 H=2187:00 +CPU= 20.840
  11:04:29 STEP 2188 H=2188:00 +CPU= 13.483
  11:04:42 STEP 2189 H=2189:00 +CPU= 13.094
  11:05:04 STEP 2190 H=2190:00 +CPU= 21.385
  11:05:18 STEP 2191 H=2191:00 +CPU= 13.074
  11:05:32 STEP 2192 H=2192:00 +CPU= 13.574
  11:05:53 STEP 2193 H=2193:00 +CPU= 20.790
  11:06:07 STEP 2194 H=2194:00 +CPU= 13.487
  11:06:21 STEP 2195 H=2195:00 +CPU= 13.196
  11:06:45 STEP 2196 H=2196:00 +CPU= 23.247
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002196
  11:06:58 STEP 2197 H=2197:00 +CPU= 12.920
  11:07:12 STEP 2198 H=2198:00 +CPU= 13.573
  11:07:33 STEP 2199 H=2199:00 +CPU= 20.746
  11:07:47 STEP 2200 H=2200:00 +CPU= 13.493
  11:08:01 STEP 2201 H=2201:00 +CPU= 13.061
  11:08:22 STEP 2202 H=2202:00 +CPU= 21.217
  11:08:36 STEP 2203 H=2203:00 +CPU= 12.949
  11:08:50 STEP 2204 H=2204:00 +CPU= 13.673
  11:09:11 STEP 2205 H=2205:00 +CPU= 20.698
  11:09:25 STEP 2206 H=2206:00 +CPU= 13.448
  11:09:39 STEP 2207 H=2207:00 +CPU= 13.436
  11:10:04 STEP 2208 H=2208:00 +CPU= 24.922
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_91.zip
Uploading the intermediate file: upload_file_91.zip
  11:10:18 STEP 2209 H=2209:00 +CPU= 13.649
  11:10:33 STEP 2210 H=2210:00 +CPU= 14.136
  11:10:55 STEP 2211 H=2211:00 +CPU= 21.373
  11:11:11 STEP 2212 H=2212:00 +CPU= 14.250
  11:11:25 STEP 2213 H=2213:00 +CPU= 13.877
  11:11:49 STEP 2214 H=2214:00 +CPU= 22.998
  11:12:05 STEP 2215 H=2215:00 +CPU= 14.812
  11:12:21 STEP 2216 H=2216:00 +CPU= 15.257
  11:12:46 STEP 2217 H=2217:00 +CPU= 22.829
  11:13:02 STEP 2218 H=2218:00 +CPU= 15.343
  11:13:18 STEP 2219 H=2219:00 +CPU= 15.462
  11:13:48 STEP 2220 H=2220:00 +CPU= 27.138
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002220
  11:14:04 STEP 2221 H=2221:00 +CPU= 15.397
  11:14:21 STEP 2222 H=2222:00 +CPU= 15.668
  11:14:47 STEP 2223 H=2223:00 +CPU= 24.356
  11:15:03 STEP 2224 H=2224:00 +CPU= 15.678
  11:15:19 STEP 2225 H=2225:00 +CPU= 14.961
  11:15:45 STEP 2226 H=2226:00 +CPU= 24.534
  11:16:01 STEP 2227 H=2227:00 +CPU= 15.066
  11:16:18 STEP 2228 H=2228:00 +CPU= 15.908
  11:16:44 STEP 2229 H=2229:00 +CPU= 24.128
  11:17:01 STEP 2230 H=2230:00 +CPU= 15.880
  11:17:18 STEP 2231 H=2231:00 +CPU= 15.958
  11:17:49 STEP 2232 H=2232:00 +CPU= 28.552
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_92.zip
Uploading the intermediate file: upload_file_92.zip
  11:18:05 STEP 2233 H=2233:00 +CPU= 15.467
  11:18:23 STEP 2234 H=2234:00 +CPU= 16.882
  11:18:51 STEP 2235 H=2235:00 +CPU= 25.621
  11:19:08 STEP 2236 H=2236:00 +CPU= 16.507
  11:19:25 STEP 2237 H=2237:00 +CPU= 16.014
  11:19:54 STEP 2238 H=2238:00 +CPU= 26.391
  11:20:12 STEP 2239 H=2239:00 +CPU= 16.415
  11:20:30 STEP 2240 H=2240:00 +CPU= 16.800
  11:20:59 STEP 2241 H=2241:00 +CPU= 25.989
  11:21:17 STEP 2242 H=2242:00 +CPU= 16.937
  11:21:35 STEP 2243 H=2243:00 +CPU= 16.457
  11:22:06 STEP 2244 H=2244:00 +CPU= 28.561
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002244
  11:22:24 STEP 2245 H=2245:00 +CPU= 16.543
  11:22:42 STEP 2246 H=2246:00 +CPU= 16.681
  11:23:10 STEP 2247 H=2247:00 +CPU= 25.980
  11:23:29 STEP 2248 H=2248:00 +CPU= 17.195
  11:23:46 STEP 2249 H=2249:00 +CPU= 15.895
  11:24:13 STEP 2250 H=2250:00 +CPU= 24.540
  11:24:29 STEP 2251 H=2251:00 +CPU= 15.460
  11:24:46 STEP 2252 H=2252:00 +CPU= 15.772
  11:25:12 STEP 2253 H=2253:00 +CPU= 24.141
  11:25:30 STEP 2254 H=2254:00 +CPU= 16.149
  11:25:47 STEP 2255 H=2255:00 +CPU= 15.842
  11:26:18 STEP 2256 H=2256:00 +CPU= 28.374
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_93.zip
Uploading the intermediate file: upload_file_93.zip
  11:26:34 STEP 2257 H=2257:00 +CPU= 15.375
  11:26:51 STEP 2258 H=2258:00 +CPU= 16.053
  11:27:17 STEP 2259 H=2259:00 +CPU= 24.226
  11:27:35 STEP 2260 H=2260:00 +CPU= 15.938
  11:27:51 STEP 2261 H=2261:00 +CPU= 15.183
  11:28:18 STEP 2262 H=2262:00 +CPU= 24.829
  11:28:34 STEP 2263 H=2263:00 +CPU= 15.609
  11:28:51 STEP 2264 H=2264:00 +CPU= 15.879
  11:29:17 STEP 2265 H=2265:00 +CPU= 24.096
  11:29:34 STEP 2266 H=2266:00 +CPU= 15.930
  11:29:51 STEP 2267 H=2267:00 +CPU= 15.643
  11:30:20 STEP 2268 H=2268:00 +CPU= 27.077
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002268
  11:30:37 STEP 2269 H=2269:00 +CPU= 15.662
  11:30:54 STEP 2270 H=2270:00 +CPU= 16.016
  11:31:19 STEP 2271 H=2271:00 +CPU= 24.131
  11:31:36 STEP 2272 H=2272:00 +CPU= 15.872
  11:31:53 STEP 2273 H=2273:00 +CPU= 15.547
  11:32:20 STEP 2274 H=2274:00 +CPU= 24.813
  11:32:36 STEP 2275 H=2275:00 +CPU= 15.400
  11:32:53 STEP 2276 H=2276:00 +CPU= 15.845
  11:33:19 STEP 2277 H=2277:00 +CPU= 24.152
  11:33:36 STEP 2278 H=2278:00 +CPU= 15.913
  11:33:52 STEP 2279 H=2279:00 +CPU= 15.761
  11:34:21 STEP 2280 H=2280:00 +CPU= 27.422
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_94.zip
Uploading the intermediate file: upload_file_94.zip
  11:34:36 STEP 2281 H=2281:00 +CPU= 14.887
  11:34:53 STEP 2282 H=2282:00 +CPU= 15.694
  11:35:17 STEP 2283 H=2283:00 +CPU= 23.482
  11:35:33 STEP 2284 H=2284:00 +CPU= 15.386
  11:35:49 STEP 2285 H=2285:00 +CPU= 14.977
  11:36:14 STEP 2286 H=2286:00 +CPU= 24.058
  11:36:30 STEP 2287 H=2287:00 +CPU= 14.961
  11:36:46 STEP 2288 H=2288:00 +CPU= 15.621
  11:37:10 STEP 2289 H=2289:00 +CPU= 23.505
  11:37:26 STEP 2290 H=2290:00 +CPU= 15.455
  11:37:42 STEP 2291 H=2291:00 +CPU= 15.190
  11:38:09 STEP 2292 H=2292:00 +CPU= 26.370
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002292
  11:38:25 STEP 2293 H=2293:00 +CPU= 15.169
  11:38:40 STEP 2294 H=2294:00 +CPU= 15.436
  11:39:03 STEP 2295 H=2295:00 +CPU= 23.100
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0904
(argv4) batchid: 950
(argv5) wuid: 12167548
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/5
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6745e5301196e3837dcf642d186f007e to: /var/lib/boinc/slots/5/oifs_43r3_ps_0904_1981050100_123_950_12167548.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/5/oifs_43r3_ps_0904_1981050100_123_950_12167548.zip
ic_ancil_file: ic_ancil_12167548
ifsdata_file: ifsdata_12167548
climate_data_file: clim_data_12167548
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0aad1e958e4aec015bd3f8e586011e6c to: /var/lib/boinc/slots/5/ic_ancil_12167548.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/5/ic_ancil_12167548.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_67b1f4a8c0a9a550b3b76ef4a3ccfb2f to: /var/lib/boinc/slots/5/ifsdata/ifsdata_12167548.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/5/ifsdata/ifsdata_12167548.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/5/159l_2/clim_data_12167548.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/5/159l_2/clim_data_12167548.zip
Checking for progress XML file: /var/lib/boinc/slots/5/progress_file_12167548.xml
Opened progress file ok : /var/lib/boinc/slots/5/progress_file_12167548.xml
-- Model is restarting --
Adjusting last_iter, 2296, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/5/progress_file_12167548.xml
last_cpu_time: 46893
upload_file_number: 95
last_iter: 2279
last_upload: 8208000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060
The child process has been launched with process id: 7439
Executing the command: /var/lib/boinc/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7439:7439] [20230122:114051:1674387651:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:41:24 STEP 2280 H=2280:00 +CPU= 26.590
  11:41:40 STEP 2281 H=2281:00 +CPU= 14.968
  11:41:57 STEP 2282 H=2282:00 +CPU= 15.964
  11:42:22 STEP 2283 H=2283:00 +CPU= 23.526
  11:42:39 STEP 2284 H=2284:00 +CPU= 15.973
  11:42:56 STEP 2285 H=2285:00 +CPU= 15.576
  11:43:21 STEP 2286 H=2286:00 +CPU= 23.910
  11:43:37 STEP 2287 H=2287:00 +CPU= 15.461
  11:43:53 STEP 2288 H=2288:00 +CPU= 15.852
  11:44:17 STEP 2289 H=2289:00 +CPU= 23.163
  11:44:33 STEP 2290 H=2290:00 +CPU= 15.688
  11:44:49 STEP 2291 H=2291:00 +CPU= 15.473
  11:45:16 STEP 2292 H=2292:00 +CPU= 25.868
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002292
  11:45:32 STEP 2293 H=2293:00 +CPU= 15.251
  11:45:48 STEP 2294 H=2294:00 +CPU= 15.669
  11:46:12 STEP 2295 H=2295:00 +CPU= 23.365
  11:46:28 STEP 2296 H=2296:00 +CPU= 15.540
  11:46:43 STEP 2297 H=2297:00 +CPU= 15.133
  11:47:08 STEP 2298 H=2298:00 +CPU= 23.792
  11:47:23 STEP 2299 H=2299:00 +CPU= 15.148
  11:47:40 STEP 2300 H=2300:00 +CPU= 15.638
  11:48:04 STEP 2301 H=2301:00 +CPU= 23.019
  11:48:20 STEP 2302 H=2302:00 +CPU= 15.479
  11:48:36 STEP 2303 H=2303:00 +CPU= 15.536
  11:49:04 STEP 2304 H=2304:00 +CPU= 26.911
Moving to projects directory: /var/lib/boinc/slots/5/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc/slots/5/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc/slots/5/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167548/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0904_1981050100_123_950_12167548_0_r149188060_95.zip
Uploading the intermediate file: upload_file_95.zip
  11:49:21 STEP 2305 H=2305:00 +CPU= 15.096
  11:49:38 STEP 2306 H=2306:00 +CPU= 15.363
  11:50:04 STEP 2307 H=2307:00 +CPU= 23.080
  11:50:20 STEP 2308 H=2308:00 +CPU= 15.522
  11:50:37 STEP 2309 H=2309:00 +CPU= 15.271
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:35:22 1531398 22251773 oifs_43r3_ps_0904_1981050100_123_950_12167548_0 7,257,600 41,562 0.0057
09 May 2023 16:35:16 1531398 22251773 oifs_43r3_ps_0904_1981050100_123_950_12167548_0 6,393,600 36,956 0.0058
09 May 2023 16:35:07 1531398 22251773 oifs_43r3_ps_0904_1981050100_123_950_12167548_0 5,529,600 32,335 0.0058
09 May 2023 16:35:00 1531398 22251773 oifs_43r3_ps_0904_1981050100_123_950_12167548_0 4,665,600 27,614 0.0059
09 May 2023 16:34:06 1531398 22251773 oifs_43r3_ps_0904_1981050100_123_950_12167548_0 2,332,800 12,909 0.0055
09 May 2023 16:33:03 1531398 22251773 oifs_43r3_ps_0904_1981050100_123_950_12167548_0 864,000 4,706 0.0054


©2024 climateprediction.net