climateprediction.net home page
Task 22268808

Task 22268808

Name oifs_43r3_ps_0973_1997050100_123_966_12183617_0
Workunit 12183617
Created 22 Dec 2022, 14:58:47 UTC
Sent 21 Jan 2023, 9:10:29 UTC
Report deadline 22 Mar 2023, 9:10:29 UTC
Received 22 Jan 2023, 12:33:38 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1531398
Run time 12 hours 55 min 32 sec
CPU time 12 hours 38 min 24 sec
Validate state Invalid
Credit 10,160.64
Device peak FLOPS 7.92 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,803.66 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,933.07 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_72.zip
Uploading the intermediate file: upload_file_72.zip
  08:33:28 STEP 1753 H=1753:00 +CPU= 15.614
  08:33:44 STEP 1754 H=1754:00 +CPU= 15.984
  08:34:08 STEP 1755 H=1755:00 +CPU= 24.339
  08:34:25 STEP 1756 H=1756:00 +CPU= 16.648
  08:34:41 STEP 1757 H=1757:00 +CPU= 16.503
  08:35:07 STEP 1758 H=1758:00 +CPU= 25.463
  08:35:22 STEP 1759 H=1759:00 +CPU= 15.414
  08:35:38 STEP 1760 H=1760:00 +CPU= 16.051
  08:36:03 STEP 1761 H=1761:00 +CPU= 24.303
  08:36:19 STEP 1762 H=1762:00 +CPU= 16.034
  08:36:35 STEP 1763 H=1763:00 +CPU= 15.833
  08:37:02 STEP 1764 H=1764:00 +CPU= 27.278
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001764
  08:37:19 STEP 1765 H=1765:00 +CPU= 16.717
  08:37:36 STEP 1766 H=1766:00 +CPU= 17.027
  08:38:01 STEP 1767 H=1767:00 +CPU= 24.909
  08:38:18 STEP 1768 H=1768:00 +CPU= 16.958
  08:38:34 STEP 1769 H=1769:00 +CPU= 16.033
  08:38:59 STEP 1770 H=1770:00 +CPU= 25.250
  08:39:15 STEP 1771 H=1771:00 +CPU= 15.450
  08:39:31 STEP 1772 H=1772:00 +CPU= 16.590
  08:39:56 STEP 1773 H=1773:00 +CPU= 24.885
  08:40:13 STEP 1774 H=1774:00 +CPU= 16.575
  08:40:29 STEP 1775 H=1775:00 +CPU= 16.241
  08:40:58 STEP 1776 H=1776:00 +CPU= 28.315
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_73.zip
Uploading the intermediate file: upload_file_73.zip
  08:41:14 STEP 1777 H=1777:00 +CPU= 16.330
Uploading trickle at timestep: 6393600
  08:41:31 STEP 1778 H=1778:00 +CPU= 17.067
  08:41:56 STEP 1779 H=1779:00 +CPU= 24.419
  08:42:12 STEP 1780 H=1780:00 +CPU= 15.941
  08:42:27 STEP 1781 H=1781:00 +CPU= 15.532
  08:42:52 STEP 1782 H=1782:00 +CPU= 24.997
  08:43:08 STEP 1783 H=1783:00 +CPU= 15.356
  08:43:24 STEP 1784 H=1784:00 +CPU= 16.318
  08:43:48 STEP 1785 H=1785:00 +CPU= 23.936
  08:44:04 STEP 1786 H=1786:00 +CPU= 15.867
  08:44:20 STEP 1787 H=1787:00 +CPU= 15.687
  08:44:47 STEP 1788 H=1788:00 +CPU= 27.598
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001788
  08:45:03 STEP 1789 H=1789:00 +CPU= 15.998
  08:45:21 STEP 1790 H=1790:00 +CPU= 16.957
  08:45:45 STEP 1791 H=1791:00 +CPU= 24.562
  08:46:02 STEP 1792 H=1792:00 +CPU= 16.562
  08:46:18 STEP 1793 H=1793:00 +CPU= 16.390
  08:46:44 STEP 1794 H=1794:00 +CPU= 25.555
  08:47:00 STEP 1795 H=1795:00 +CPU= 16.175
  08:47:17 STEP 1796 H=1796:00 +CPU= 17.010
  08:47:41 STEP 1797 H=1797:00 +CPU= 24.201
  08:47:57 STEP 1798 H=1798:00 +CPU= 15.721
  08:48:13 STEP 1799 H=1799:00 +CPU= 15.986
  08:48:41 STEP 1800 H=1800:00 +CPU= 28.094
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_74.zip
Uploading the intermediate file: upload_file_74.zip
  08:48:57 STEP 1801 H=1801:00 +CPU= 15.515
  08:49:14 STEP 1802 H=1802:00 +CPU= 16.817
  08:49:38 STEP 1803 H=1803:00 +CPU= 23.919
  08:49:54 STEP 1804 H=1804:00 +CPU= 15.788
  08:50:12 STEP 1805 H=1805:00 +CPU= 16.352
  08:50:37 STEP 1806 H=1806:00 +CPU= 25.452
  08:50:54 STEP 1807 H=1807:00 +CPU= 16.320
  08:51:11 STEP 1808 H=1808:00 +CPU= 17.070
  08:51:35 STEP 1809 H=1809:00 +CPU= 23.874
  08:51:51 STEP 1810 H=1810:00 +CPU= 16.244
  08:52:07 STEP 1811 H=1811:00 +CPU= 16.535
  08:52:34 STEP 1812 H=1812:00 +CPU= 26.783
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001812
  08:52:50 STEP 1813 H=1813:00 +CPU= 15.552
  08:53:06 STEP 1814 H=1814:00 +CPU= 15.907
  08:53:30 STEP 1815 H=1815:00 +CPU= 23.679
  08:53:46 STEP 1816 H=1816:00 +CPU= 15.991
  08:54:01 STEP 1817 H=1817:00 +CPU= 15.530
  08:54:26 STEP 1818 H=1818:00 +CPU= 24.725
  08:54:42 STEP 1819 H=1819:00 +CPU= 16.156
  08:54:59 STEP 1820 H=1820:00 +CPU= 16.550
  08:55:23 STEP 1821 H=1821:00 +CPU= 24.021
  08:55:39 STEP 1822 H=1822:00 +CPU= 16.024
  08:55:55 STEP 1823 H=1823:00 +CPU= 16.171
  08:56:23 STEP 1824 H=1824:00 +CPU= 28.120
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_75.zip
Uploading the intermediate file: upload_file_75.zip
  08:56:39 STEP 1825 H=1825:00 +CPU= 15.874
  08:56:55 STEP 1826 H=1826:00 +CPU= 15.979
  08:57:19 STEP 1827 H=1827:00 +CPU= 24.120
  08:57:36 STEP 1828 H=1828:00 +CPU= 16.060
  08:57:52 STEP 1829 H=1829:00 +CPU= 16.467
  08:58:17 STEP 1830 H=1830:00 +CPU= 25.146
  08:58:33 STEP 1831 H=1831:00 +CPU= 15.358
  08:58:49 STEP 1832 H=1832:00 +CPU= 16.293
  08:59:13 STEP 1833 H=1833:00 +CPU= 24.144
  08:59:29 STEP 1834 H=1834:00 +CPU= 16.047
  08:59:45 STEP 1835 H=1835:00 +CPU= 15.708
  09:00:12 STEP 1836 H=1836:00 +CPU= 27.300
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001836
  09:00:28 STEP 1837 H=1837:00 +CPU= 15.475
  09:00:44 STEP 1838 H=1838:00 +CPU= 16.323
  09:01:09 STEP 1839 H=1839:00 +CPU= 25.304
  09:01:26 STEP 1840 H=1840:00 +CPU= 16.974
  09:01:42 STEP 1841 H=1841:00 +CPU= 15.708
  09:02:07 STEP 1842 H=1842:00 +CPU= 24.549
  09:02:23 STEP 1843 H=1843:00 +CPU= 15.844
  09:02:39 STEP 1844 H=1844:00 +CPU= 16.307
  09:03:03 STEP 1845 H=1845:00 +CPU= 24.084
  09:03:19 STEP 1846 H=1846:00 +CPU= 15.918
  09:03:35 STEP 1847 H=1847:00 +CPU= 16.307
  09:04:03 STEP 1848 H=1848:00 +CPU= 27.909
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_76.zip
Uploading the intermediate file: upload_file_76.zip
  09:04:19 STEP 1849 H=1849:00 +CPU= 15.159
  09:04:35 STEP 1850 H=1850:00 +CPU= 16.223
  09:05:00 STEP 1851 H=1851:00 +CPU= 24.585
  09:05:16 STEP 1852 H=1852:00 +CPU= 16.737
  09:05:33 STEP 1853 H=1853:00 +CPU= 16.505
  09:05:58 STEP 1854 H=1854:00 +CPU= 25.318
  09:06:14 STEP 1855 H=1855:00 +CPU= 16.068
  09:06:31 STEP 1856 H=1856:00 +CPU= 16.783
  09:06:55 STEP 1857 H=1857:00 +CPU= 24.064
  09:07:12 STEP 1858 H=1858:00 +CPU= 17.038
  09:07:29 STEP 1859 H=1859:00 +CPU= 16.775
  09:07:57 STEP 1860 H=1860:00 +CPU= 27.318
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001860
  09:08:13 STEP 1861 H=1861:00 +CPU= 16.754
  09:08:30 STEP 1862 H=1862:00 +CPU= 16.294
  09:08:54 STEP 1863 H=1863:00 +CPU= 24.042
  09:09:11 STEP 1864 H=1864:00 +CPU= 16.875
  09:09:27 STEP 1865 H=1865:00 +CPU= 16.371
  09:09:52 STEP 1866 H=1866:00 +CPU= 24.385
  09:10:08 STEP 1867 H=1867:00 +CPU= 16.097
  09:10:24 STEP 1868 H=1868:00 +CPU= 16.498
  09:10:49 STEP 1869 H=1869:00 +CPU= 25.078
  09:11:06 STEP 1870 H=1870:00 +CPU= 16.504
  09:11:22 STEP 1871 H=1871:00 +CPU= 15.968
  09:11:51 STEP 1872 H=1872:00 +CPU= 28.490
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_77.zip
Uploading the intermediate file: upload_file_77.zip
  09:12:07 STEP 1873 H=1873:00 +CPU= 16.055
  09:12:24 STEP 1874 H=1874:00 +CPU= 17.230
  09:12:49 STEP 1875 H=1875:00 +CPU= 24.795
  09:13:05 STEP 1876 H=1876:00 +CPU= 16.338
  09:13:22 STEP 1877 H=1877:00 +CPU= 16.424
  09:13:47 STEP 1878 H=1878:00 +CPU= 24.769
  09:14:03 STEP 1879 H=1879:00 +CPU= 16.474
  09:14:19 STEP 1880 H=1880:00 +CPU= 16.445
  09:14:43 STEP 1881 H=1881:00 +CPU= 23.830
  09:15:00 STEP 1882 H=1882:00 +CPU= 16.153
  09:15:15 STEP 1883 H=1883:00 +CPU= 15.458
  09:15:42 STEP 1884 H=1884:00 +CPU= 26.551
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001884
  09:15:57 STEP 1885 H=1885:00 +CPU= 15.542
  09:16:14 STEP 1886 H=1886:00 +CPU= 16.683
  09:16:39 STEP 1887 H=1887:00 +CPU= 24.577
  09:16:55 STEP 1888 H=1888:00 +CPU= 16.821
  09:17:12 STEP 1889 H=1889:00 +CPU= 16.570
  09:17:38 STEP 1890 H=1890:00 +CPU= 25.597
  09:17:54 STEP 1891 H=1891:00 +CPU= 16.370
  09:18:11 STEP 1892 H=1892:00 +CPU= 16.681
  09:18:35 STEP 1893 H=1893:00 +CPU= 24.048
  09:18:51 STEP 1894 H=1894:00 +CPU= 16.010
  09:19:07 STEP 1895 H=1895:00 +CPU= 16.540
  09:19:36 STEP 1896 H=1896:00 +CPU= 28.881
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_78.zip
Uploading the intermediate file: upload_file_78.zip
  09:19:52 STEP 1897 H=1897:00 +CPU= 15.524
  09:20:08 STEP 1898 H=1898:00 +CPU= 16.259
  09:20:33 STEP 1899 H=1899:00 +CPU= 24.351
  09:20:50 STEP 1900 H=1900:00 +CPU= 16.898
  09:21:06 STEP 1901 H=1901:00 +CPU= 15.910
  09:21:30 STEP 1902 H=1902:00 +CPU= 24.528
  09:21:46 STEP 1903 H=1903:00 +CPU= 15.673
  09:22:02 STEP 1904 H=1904:00 +CPU= 16.423
  09:22:27 STEP 1905 H=1905:00 +CPU= 25.004
  09:22:44 STEP 1906 H=1906:00 +CPU= 16.829
  09:23:00 STEP 1907 H=1907:00 +CPU= 15.793
  09:23:26 STEP 1908 H=1908:00 +CPU= 26.217
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001908
  09:23:42 STEP 1909 H=1909:00 +CPU= 15.564
  09:23:58 STEP 1910 H=1910:00 +CPU= 16.174
  09:24:23 STEP 1911 H=1911:00 +CPU= 24.448
  09:24:39 STEP 1912 H=1912:00 +CPU= 15.969
  09:24:54 STEP 1913 H=1913:00 +CPU= 15.472
  09:25:19 STEP 1914 H=1914:00 +CPU= 24.849
  09:25:34 STEP 1915 H=1915:00 +CPU= 15.482
  09:25:51 STEP 1916 H=1916:00 +CPU= 16.119
  09:26:15 STEP 1917 H=1917:00 +CPU= 24.539
  09:26:32 STEP 1918 H=1918:00 +CPU= 16.849
  09:26:49 STEP 1919 H=1919:00 +CPU= 16.806
  09:27:18 STEP 1920 H=1920:00 +CPU= 29.148
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_79.zip
Uploading the intermediate file: upload_file_79.zip
  09:27:34 STEP 1921 H=1921:00 +CPU= 16.067
  09:27:51 STEP 1922 H=1922:00 +CPU= 16.750
  09:28:16 STEP 1923 H=1923:00 +CPU= 24.929
  09:28:33 STEP 1924 H=1924:00 +CPU= 16.647
  09:28:49 STEP 1925 H=1925:00 +CPU= 16.052
  09:29:13 STEP 1926 H=1926:00 +CPU= 24.366
  09:29:29 STEP 1927 H=1927:00 +CPU= 15.652
  09:29:45 STEP 1928 H=1928:00 +CPU= 16.014
  09:30:09 STEP 1929 H=1929:00 +CPU= 24.138
  09:30:26 STEP 1930 H=1930:00 +CPU= 16.138
  09:30:41 STEP 1931 H=1931:00 +CPU= 15.678
  09:31:08 STEP 1932 H=1932:00 +CPU= 26.548
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001932
  09:31:24 STEP 1933 H=1933:00 +CPU= 15.709
  09:31:40 STEP 1934 H=1934:00 +CPU= 16.731
  09:32:05 STEP 1935 H=1935:00 +CPU= 24.437
  09:32:21 STEP 1936 H=1936:00 +CPU= 16.115
  09:32:37 STEP 1937 H=1937:00 +CPU= 15.667
  09:33:01 STEP 1938 H=1938:00 +CPU= 24.579
  09:33:17 STEP 1939 H=1939:00 +CPU= 15.606
  09:33:33 STEP 1940 H=1940:00 +CPU= 16.213
  09:33:57 STEP 1941 H=1941:00 +CPU= 23.960
  09:34:13 STEP 1942 H=1942:00 +CPU= 16.034
  09:34:29 STEP 1943 H=1943:00 +CPU= 15.997
  09:34:58 STEP 1944 H=1944:00 +CPU= 28.443
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_80.zip
Uploading the intermediate file: upload_file_80.zip
  09:35:14 STEP 1945 H=1945:00 +CPU= 15.597
  09:35:30 STEP 1946 H=1946:00 +CPU= 16.356
  09:35:55 STEP 1947 H=1947:00 +CPU= 24.679
  09:36:12 STEP 1948 H=1948:00 +CPU= 16.845
  09:36:28 STEP 1949 H=1949:00 +CPU= 16.124
  09:36:53 STEP 1950 H=1950:00 +CPU= 25.391
  09:37:10 STEP 1951 H=1951:00 +CPU= 16.462
  09:37:26 STEP 1952 H=1952:00 +CPU= 16.600
  09:37:51 STEP 1953 H=1953:00 +CPU= 24.617
  09:38:07 STEP 1954 H=1954:00 +CPU= 16.325
  09:38:23 STEP 1955 H=1955:00 +CPU= 15.184
  09:38:49 STEP 1956 H=1956:00 +CPU= 26.443
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001956
  09:39:05 STEP 1957 H=1957:00 +CPU= 15.771
  09:39:22 STEP 1958 H=1958:00 +CPU= 16.663
  09:39:46 STEP 1959 H=1959:00 +CPU= 24.552
  09:40:02 STEP 1960 H=1960:00 +CPU= 16.160
  09:40:18 STEP 1961 H=1961:00 +CPU= 15.363
  09:40:42 STEP 1962 H=1962:00 +CPU= 24.669
  09:40:59 STEP 1963 H=1963:00 +CPU= 16.468
  09:41:16 STEP 1964 H=1964:00 +CPU= 16.814
  09:41:41 STEP 1965 H=1965:00 +CPU= 24.904
  09:41:58 STEP 1966 H=1966:00 +CPU= 16.869
  09:42:15 STEP 1967 H=1967:00 +CPU= 16.945
  09:42:43 STEP 1968 H=1968:00 +CPU= 28.600
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_81.zip
Uploading the intermediate file: upload_file_81.zip
  09:42:59 STEP 1969 H=1969:00 +CPU= 15.513
  09:43:15 STEP 1970 H=1970:00 +CPU= 16.289
  09:43:39 STEP 1971 H=1971:00 +CPU= 23.920
  09:43:56 STEP 1972 H=1972:00 +CPU= 16.054
  09:44:11 STEP 1973 H=1973:00 +CPU= 15.512
  09:44:35 STEP 1974 H=1974:00 +CPU= 24.275
  09:44:51 STEP 1975 H=1975:00 +CPU= 15.508
  09:45:07 STEP 1976 H=1976:00 +CPU= 16.034
  09:45:31 STEP 1977 H=1977:00 +CPU= 24.085
  09:45:47 STEP 1978 H=1978:00 +CPU= 15.800
  09:46:02 STEP 1979 H=1979:00 +CPU= 15.318
  09:46:29 STEP 1980 H=1980:00 +CPU= 27.066
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001980
  09:46:45 STEP 1981 H=1981:00 +CPU= 16.028
  09:47:02 STEP 1982 H=1982:00 +CPU= 16.446
  09:47:27 STEP 1983 H=1983:00 +CPU= 24.517
  09:47:43 STEP 1984 H=1984:00 +CPU= 15.959
  09:47:58 STEP 1985 H=1985:00 +CPU= 15.531
  09:48:23 STEP 1986 H=1986:00 +CPU= 25.106
  09:48:39 STEP 1987 H=1987:00 +CPU= 15.629
  09:48:56 STEP 1988 H=1988:00 +CPU= 16.838
  09:49:21 STEP 1989 H=1989:00 +CPU= 25.076
  09:49:38 STEP 1990 H=1990:00 +CPU= 16.809
  09:49:54 STEP 1991 H=1991:00 +CPU= 16.632
  09:50:23 STEP 1992 H=1992:00 +CPU= 28.950
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_82.zip
Uploading the intermediate file: upload_file_82.zip
  09:50:40 STEP 1993 H=1993:00 +CPU= 16.263
  09:50:57 STEP 1994 H=1994:00 +CPU= 16.864
  09:51:21 STEP 1995 H=1995:00 +CPU= 24.080
  09:51:37 STEP 1996 H=1996:00 +CPU= 16.116
  09:51:53 STEP 1997 H=1997:00 +CPU= 15.675
  09:52:17 STEP 1998 H=1998:00 +CPU= 24.355
  09:52:33 STEP 1999 H=1999:00 +CPU= 15.623
  09:52:49 STEP 2000 H=2000:00 +CPU= 15.937
  09:53:13 STEP 2001 H=2001:00 +CPU= 23.840
  09:53:29 STEP 2002 H=2002:00 +CPU= 16.004
  09:53:44 STEP 2003 H=2003:00 +CPU= 15.577
  09:54:11 STEP 2004 H=2004:00 +CPU= 27.198
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002004
  09:54:28 STEP 2005 H=2005:00 +CPU= 16.395
  09:54:44 STEP 2006 H=2006:00 +CPU= 16.341
  09:55:10 STEP 2007 H=2007:00 +CPU= 25.296
  09:55:26 STEP 2008 H=2008:00 +CPU= 16.869
  09:55:43 STEP 2009 H=2009:00 +CPU= 16.092
  09:56:08 STEP 2010 H=2010:00 +CPU= 25.787
  09:56:24 STEP 2011 H=2011:00 +CPU= 15.496
  09:56:40 STEP 2012 H=2012:00 +CPU= 15.934
  09:57:04 STEP 2013 H=2013:00 +CPU= 23.774
  09:57:20 STEP 2014 H=2014:00 +CPU= 15.871
  09:57:36 STEP 2015 H=2015:00 +CPU= 16.244
  09:58:04 STEP 2016 H=2016:00 +CPU= 28.500
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_83.zip
Uploading the intermediate file: upload_file_83.zip
  09:58:20 STEP 2017 H=2017:00 +CPU= 15.859
Uploading trickle at timestep: 7257600
  09:58:37 STEP 2018 H=2018:00 +CPU= 16.280
  09:59:01 STEP 2019 H=2019:00 +CPU= 24.121
  09:59:17 STEP 2020 H=2020:00 +CPU= 15.899
  09:59:33 STEP 2021 H=2021:00 +CPU= 15.587
  09:59:57 STEP 2022 H=2022:00 +CPU= 24.516
  10:00:13 STEP 2023 H=2023:00 +CPU= 15.504
  10:00:29 STEP 2024 H=2024:00 +CPU= 16.126
  10:00:53 STEP 2025 H=2025:00 +CPU= 24.333
  10:01:10 STEP 2026 H=2026:00 +CPU= 16.382
  10:01:25 STEP 2027 H=2027:00 +CPU= 15.728
  10:01:53 STEP 2028 H=2028:00 +CPU= 27.587
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002028
  10:02:09 STEP 2029 H=2029:00 +CPU= 15.585
  10:02:25 STEP 2030 H=2030:00 +CPU= 16.393
  10:02:49 STEP 2031 H=2031:00 +CPU= 23.932
  10:03:05 STEP 2032 H=2032:00 +CPU= 15.968
  10:03:21 STEP 2033 H=2033:00 +CPU= 15.801
  10:03:46 STEP 2034 H=2034:00 +CPU= 25.502
  10:04:02 STEP 2035 H=2035:00 +CPU= 16.002
  10:04:19 STEP 2036 H=2036:00 +CPU= 16.518
  10:04:43 STEP 2037 H=2037:00 +CPU= 24.363
  10:04:59 STEP 2038 H=2038:00 +CPU= 16.017
  10:05:16 STEP 2039 H=2039:00 +CPU= 16.198
  10:05:44 STEP 2040 H=2040:00 +CPU= 28.257
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_84.zip
Uploading the intermediate file: upload_file_84.zip
  10:06:00 STEP 2041 H=2041:00 +CPU= 16.103
  10:06:17 STEP 2042 H=2042:00 +CPU= 16.787
  10:06:41 STEP 2043 H=2043:00 +CPU= 23.914
  10:06:57 STEP 2044 H=2044:00 +CPU= 16.206
  10:07:13 STEP 2045 H=2045:00 +CPU= 16.105
  10:07:39 STEP 2046 H=2046:00 +CPU= 25.660
  10:07:55 STEP 2047 H=2047:00 +CPU= 16.188
  10:08:12 STEP 2048 H=2048:00 +CPU= 16.798
  10:08:37 STEP 2049 H=2049:00 +CPU= 24.710
  10:08:53 STEP 2050 H=2050:00 +CPU= 15.647
  10:09:08 STEP 2051 H=2051:00 +CPU= 15.546
  10:09:35 STEP 2052 H=2052:00 +CPU= 26.626
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002052
  10:09:50 STEP 2053 H=2053:00 +CPU= 15.334
  10:10:07 STEP 2054 H=2054:00 +CPU= 16.689
  10:10:31 STEP 2055 H=2055:00 +CPU= 24.167
  10:10:47 STEP 2056 H=2056:00 +CPU= 16.052
  10:11:03 STEP 2057 H=2057:00 +CPU= 15.617
  10:11:28 STEP 2058 H=2058:00 +CPU= 24.930
  10:11:43 STEP 2059 H=2059:00 +CPU= 15.418
  10:12:00 STEP 2060 H=2060:00 +CPU= 16.477
  10:12:24 STEP 2061 H=2061:00 +CPU= 24.080
  10:12:40 STEP 2062 H=2062:00 +CPU= 15.702
  10:12:56 STEP 2063 H=2063:00 +CPU= 16.106
  10:13:25 STEP 2064 H=2064:00 +CPU= 29.204
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_85.zip
Uploading the intermediate file: upload_file_85.zip
  10:13:41 STEP 2065 H=2065:00 +CPU= 15.947
  10:13:58 STEP 2066 H=2066:00 +CPU= 16.778
  10:14:23 STEP 2067 H=2067:00 +CPU= 25.241
  10:14:40 STEP 2068 H=2068:00 +CPU= 16.514
  10:14:56 STEP 2069 H=2069:00 +CPU= 16.285
  10:15:22 STEP 2070 H=2070:00 +CPU= 25.262
  10:15:37 STEP 2071 H=2071:00 +CPU= 15.397
  10:15:54 STEP 2072 H=2072:00 +CPU= 16.926
  10:16:19 STEP 2073 H=2073:00 +CPU= 25.073
  10:16:36 STEP 2074 H=2074:00 +CPU= 16.609
  10:16:52 STEP 2075 H=2075:00 +CPU= 16.601
  10:17:20 STEP 2076 H=2076:00 +CPU= 27.748
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002076
  10:17:36 STEP 2077 H=2077:00 +CPU= 15.931
  10:17:52 STEP 2078 H=2078:00 +CPU= 16.216
  10:18:17 STEP 2079 H=2079:00 +CPU= 24.913
  10:18:34 STEP 2080 H=2080:00 +CPU= 16.664
  10:18:50 STEP 2081 H=2081:00 +CPU= 15.495
  10:19:14 STEP 2082 H=2082:00 +CPU= 24.661
  10:19:30 STEP 2083 H=2083:00 +CPU= 15.382
  10:19:46 STEP 2084 H=2084:00 +CPU= 15.960
  10:20:10 STEP 2085 H=2085:00 +CPU= 24.097
  10:20:27 STEP 2086 H=2086:00 +CPU= 16.683
  10:20:43 STEP 2087 H=2087:00 +CPU= 16.687
  10:21:12 STEP 2088 H=2088:00 +CPU= 28.924
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_86.zip
Uploading the intermediate file: upload_file_86.zip
  10:21:29 STEP 2089 H=2089:00 +CPU= 16.038
  10:21:46 STEP 2090 H=2090:00 +CPU= 17.100
  10:22:10 STEP 2091 H=2091:00 +CPU= 24.187
  10:22:26 STEP 2092 H=2092:00 +CPU= 15.851
  10:22:42 STEP 2093 H=2093:00 +CPU= 16.214
  10:23:08 STEP 2094 H=2094:00 +CPU= 25.793
  10:23:24 STEP 2095 H=2095:00 +CPU= 16.173
  10:23:41 STEP 2096 H=2096:00 +CPU= 16.727
  10:24:06 STEP 2097 H=2097:00 +CPU= 24.851
  10:24:23 STEP 2098 H=2098:00 +CPU= 16.927
  10:24:39 STEP 2099 H=2099:00 +CPU= 16.408
  10:25:07 STEP 2100 H=2100:00 +CPU= 26.964
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002100
  10:25:22 STEP 2101 H=2101:00 +CPU= 15.821
  10:25:39 STEP 2102 H=2102:00 +CPU= 16.102
  10:26:02 STEP 2103 H=2103:00 +CPU= 23.545
  10:26:18 STEP 2104 H=2104:00 +CPU= 16.099
  10:26:34 STEP 2105 H=2105:00 +CPU= 15.875
  10:26:59 STEP 2106 H=2106:00 +CPU= 24.859
  10:27:16 STEP 2107 H=2107:00 +CPU= 16.619
  10:27:32 STEP 2108 H=2108:00 +CPU= 16.215
  10:27:56 STEP 2109 H=2109:00 +CPU= 23.634
  10:28:12 STEP 2110 H=2110:00 +CPU= 16.109
  10:28:29 STEP 2111 H=2111:00 +CPU= 16.622
  10:28:58 STEP 2112 H=2112:00 +CPU= 28.533
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_87.zip
Uploading the intermediate file: upload_file_87.zip
  10:29:10 STEP 2113 H=2113:00 +CPU= 11.255
  10:29:18 STEP 2114 H=2114:00 +CPU=  8.893
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0973
(argv4) batchid: 966
(argv5) wuid: 12183617
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/11
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6c6e9a87e5aa9bcc5bd10e1c1efe952d to: /var/lib/boinc/slots/11/oifs_43r3_ps_0973_1997050100_123_966_12183617.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/11/oifs_43r3_ps_0973_1997050100_123_966_12183617.zip
ic_ancil_file: ic_ancil_12183617
ifsdata_file: ifsdata_12183617
climate_data_file: clim_data_12183617
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5cecbb64fdfe5bf34e3cfc99a505ce16 to: /var/lib/boinc/slots/11/ic_ancil_12183617.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/11/ic_ancil_12183617.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e07317318b2afce857ec29a0723b15b0 to: /var/lib/boinc/slots/11/ifsdata/ifsdata_12183617.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/11/ifsdata/ifsdata_12183617.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/11/159l_2/clim_data_12183617.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/11/159l_2/clim_data_12183617.zip
Checking for progress XML file: /var/lib/boinc/slots/11/progress_file_12183617.xml
Opened progress file ok : /var/lib/boinc/slots/11/progress_file_12183617.xml
-- Model is restarting --
Adjusting last_iter, 2113, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/11/progress_file_12183617.xml
last_cpu_time: 43285
upload_file_number: 88
last_iter: 2111
last_upload: 7603200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696
The child process has been launched with process id: 2444
Executing the command: /var/lib/boinc/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:2444:2444] [20230122:110949:1674385789:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:10:15 STEP 2112 H=2112:00 +CPU= 23.991
  11:10:30 STEP 2113 H=2113:00 +CPU= 13.832
  11:10:44 STEP 2114 H=2114:00 +CPU= 14.252
  11:11:08 STEP 2115 H=2115:00 +CPU= 21.559
  11:11:23 STEP 2116 H=2116:00 +CPU= 14.300
  11:11:37 STEP 2117 H=2117:00 +CPU= 13.707
  11:12:02 STEP 2118 H=2118:00 +CPU= 23.233
  11:12:18 STEP 2119 H=2119:00 +CPU= 15.059
  11:12:33 STEP 2120 H=2120:00 +CPU= 14.983
  11:12:58 STEP 2121 H=2121:00 +CPU= 23.254
  11:13:15 STEP 2122 H=2122:00 +CPU= 15.960
  11:13:32 STEP 2123 H=2123:00 +CPU= 15.847
  11:14:02 STEP 2124 H=2124:00 +CPU= 27.260
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002124
  11:14:18 STEP 2125 H=2125:00 +CPU= 15.355
  11:14:35 STEP 2126 H=2126:00 +CPU= 16.197
  11:15:01 STEP 2127 H=2127:00 +CPU= 24.075
  11:15:17 STEP 2128 H=2128:00 +CPU= 15.501
  11:15:33 STEP 2129 H=2129:00 +CPU= 15.291
  11:15:59 STEP 2130 H=2130:00 +CPU= 24.559
  11:16:16 STEP 2131 H=2131:00 +CPU= 15.505
  11:16:33 STEP 2132 H=2132:00 +CPU= 16.292
  11:16:58 STEP 2133 H=2133:00 +CPU= 24.111
  11:17:15 STEP 2134 H=2134:00 +CPU= 15.928
  11:17:33 STEP 2135 H=2135:00 +CPU= 16.333
  11:18:03 STEP 2136 H=2136:00 +CPU= 28.229
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_88.zip
Uploading the intermediate file: upload_file_88.zip
  11:18:20 STEP 2137 H=2137:00 +CPU= 16.050
  11:18:39 STEP 2138 H=2138:00 +CPU= 17.225
  11:19:06 STEP 2139 H=2139:00 +CPU= 25.811
  11:19:23 STEP 2140 H=2140:00 +CPU= 16.301
  11:19:41 STEP 2141 H=2141:00 +CPU= 16.362
  11:20:10 STEP 2142 H=2142:00 +CPU= 27.071
  11:20:28 STEP 2143 H=2143:00 +CPU= 16.379
  11:20:47 STEP 2144 H=2144:00 +CPU= 17.564
  11:21:15 STEP 2145 H=2145:00 +CPU= 25.948
  11:21:33 STEP 2146 H=2146:00 +CPU= 17.079
  11:21:52 STEP 2147 H=2147:00 +CPU= 16.825
  11:22:22 STEP 2148 H=2148:00 +CPU= 28.433
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002148
  11:22:40 STEP 2149 H=2149:00 +CPU= 16.563
  11:22:59 STEP 2150 H=2150:00 +CPU= 16.896
  11:23:27 STEP 2151 H=2151:00 +CPU= 25.716
  11:23:45 STEP 2152 H=2152:00 +CPU= 16.725
  11:24:01 STEP 2153 H=2153:00 +CPU= 15.482
  11:24:27 STEP 2154 H=2154:00 +CPU= 24.757
  11:24:44 STEP 2155 H=2155:00 +CPU= 15.436
  11:25:01 STEP 2156 H=2156:00 +CPU= 16.091
  11:25:27 STEP 2157 H=2157:00 +CPU= 24.012
  11:25:44 STEP 2158 H=2158:00 +CPU= 15.951
  11:26:02 STEP 2159 H=2159:00 +CPU= 16.433
  11:26:33 STEP 2160 H=2160:00 +CPU= 28.356
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_89.zip
Uploading the intermediate file: upload_file_89.zip
  11:26:50 STEP 2161 H=2161:00 +CPU= 15.709
  11:27:07 STEP 2162 H=2162:00 +CPU= 16.140
  11:27:33 STEP 2163 H=2163:00 +CPU= 23.950
  11:27:50 STEP 2164 H=2164:00 +CPU= 16.280
  11:28:07 STEP 2165 H=2165:00 +CPU= 15.612
  11:28:34 STEP 2166 H=2166:00 +CPU= 24.947
  11:28:51 STEP 2167 H=2167:00 +CPU= 15.551
  11:29:08 STEP 2168 H=2168:00 +CPU= 16.268
  11:29:35 STEP 2169 H=2169:00 +CPU= 24.434
  11:29:52 STEP 2170 H=2170:00 +CPU= 16.326
  11:30:09 STEP 2171 H=2171:00 +CPU= 15.661
  11:30:38 STEP 2172 H=2172:00 +CPU= 26.643
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002172
  11:30:54 STEP 2173 H=2173:00 +CPU= 15.481
  11:31:11 STEP 2174 H=2174:00 +CPU= 16.173
  11:31:37 STEP 2175 H=2175:00 +CPU= 24.159
  11:31:55 STEP 2176 H=2176:00 +CPU= 16.368
  11:32:11 STEP 2177 H=2177:00 +CPU= 15.637
  11:32:38 STEP 2178 H=2178:00 +CPU= 24.534
  11:32:55 STEP 2179 H=2179:00 +CPU= 15.708
  11:33:12 STEP 2180 H=2180:00 +CPU= 16.106
  11:33:37 STEP 2181 H=2181:00 +CPU= 23.774
  11:33:54 STEP 2182 H=2182:00 +CPU= 15.869
  11:34:11 STEP 2183 H=2183:00 +CPU= 15.847
  11:34:39 STEP 2184 H=2184:00 +CPU= 27.279
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_90.zip
Uploading the intermediate file: upload_file_90.zip
  11:34:54 STEP 2185 H=2185:00 +CPU= 14.839
  11:35:10 STEP 2186 H=2186:00 +CPU= 15.637
  11:35:34 STEP 2187 H=2187:00 +CPU= 23.352
  11:35:51 STEP 2188 H=2188:00 +CPU= 15.628
  11:36:07 STEP 2189 H=2189:00 +CPU= 15.058
  11:36:32 STEP 2190 H=2190:00 +CPU= 24.152
  11:36:48 STEP 2191 H=2191:00 +CPU= 15.071
  11:37:04 STEP 2192 H=2192:00 +CPU= 15.311
  11:37:28 STEP 2193 H=2193:00 +CPU= 23.433
  11:37:45 STEP 2194 H=2194:00 +CPU= 15.560
  11:38:01 STEP 2195 H=2195:00 +CPU= 15.171
  11:38:27 STEP 2196 H=2196:00 +CPU= 25.868
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002196
  11:38:43 STEP 2197 H=2197:00 +CPU= 15.400
  11:38:58 STEP 2198 H=2198:00 +CPU= 15.300
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0973
(argv4) batchid: 966
(argv5) wuid: 12183617
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/11
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6c6e9a87e5aa9bcc5bd10e1c1efe952d to: /var/lib/boinc/slots/11/oifs_43r3_ps_0973_1997050100_123_966_12183617.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/11/oifs_43r3_ps_0973_1997050100_123_966_12183617.zip
ic_ancil_file: ic_ancil_12183617
ifsdata_file: ifsdata_12183617
climate_data_file: clim_data_12183617
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5cecbb64fdfe5bf34e3cfc99a505ce16 to: /var/lib/boinc/slots/11/ic_ancil_12183617.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/11/ic_ancil_12183617.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e07317318b2afce857ec29a0723b15b0 to: /var/lib/boinc/slots/11/ifsdata/ifsdata_12183617.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/11/ifsdata/ifsdata_12183617.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/11/159l_2/clim_data_12183617.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/11/159l_2/clim_data_12183617.zip
Checking for progress XML file: /var/lib/boinc/slots/11/progress_file_12183617.xml
Opened progress file ok : /var/lib/boinc/slots/11/progress_file_12183617.xml
-- Model is restarting --
Adjusting last_iter, 2199, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/11/progress_file_12183617.xml
last_cpu_time: 44938
upload_file_number: 91
last_iter: 2183
last_upload: 7862400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696
The child process has been launched with process id: 7435
Executing the command: /var/lib/boinc/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7435:7435] [20230122:114051:1674387651:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:41:23 STEP 2184 H=2184:00 +CPU= 26.450
  11:41:39 STEP 2185 H=2185:00 +CPU= 15.158
  11:41:57 STEP 2186 H=2186:00 +CPU= 16.005
  11:42:22 STEP 2187 H=2187:00 +CPU= 23.798
  11:42:39 STEP 2188 H=2188:00 +CPU= 16.181
  11:42:56 STEP 2189 H=2189:00 +CPU= 15.434
  11:43:21 STEP 2190 H=2190:00 +CPU= 24.050
  11:43:38 STEP 2191 H=2191:00 +CPU= 15.551
  11:43:54 STEP 2192 H=2192:00 +CPU= 15.601
  11:44:18 STEP 2193 H=2193:00 +CPU= 23.299
  11:44:34 STEP 2194 H=2194:00 +CPU= 15.449
  11:44:49 STEP 2195 H=2195:00 +CPU= 15.287
  11:45:17 STEP 2196 H=2196:00 +CPU= 26.569
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002196
  11:45:32 STEP 2197 H=2197:00 +CPU= 15.347
  11:45:48 STEP 2198 H=2198:00 +CPU= 15.560
  11:46:13 STEP 2199 H=2199:00 +CPU= 23.775
  11:46:29 STEP 2200 H=2200:00 +CPU= 15.580
  11:46:45 STEP 2201 H=2201:00 +CPU= 15.397
  11:47:09 STEP 2202 H=2202:00 +CPU= 24.066
  11:47:25 STEP 2203 H=2203:00 +CPU= 15.302
  11:47:42 STEP 2204 H=2204:00 +CPU= 15.481
  11:48:06 STEP 2205 H=2205:00 +CPU= 23.249
  11:48:22 STEP 2206 H=2206:00 +CPU= 15.737
  11:48:38 STEP 2207 H=2207:00 +CPU= 15.390
  11:49:07 STEP 2208 H=2208:00 +CPU= 28.016
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183617/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0973_1997050100_123_966_12183617_0_r47997696_91.zip
Uploading the intermediate file: upload_file_91.zip
  11:49:23 STEP 2209 H=2209:00 +CPU= 15.365
  11:49:41 STEP 2210 H=2210:00 +CPU= 15.481
  11:50:07 STEP 2211 H=2211:00 +CPU= 23.355
  11:50:23 STEP 2212 H=2212:00 +CPU= 15.611
  11:50:40 STEP 2213 H=2213:00 +CPU= 15.227
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:35:22 1531398 22268808 oifs_43r3_ps_0973_1997050100_123_966_12183617_0 7,257,600 41,438 0.0057
09 May 2023 16:35:16 1531398 22268808 oifs_43r3_ps_0973_1997050100_123_966_12183617_0 6,393,600 36,819 0.0058
09 May 2023 16:35:07 1531398 22268808 oifs_43r3_ps_0973_1997050100_123_966_12183617_0 5,529,600 32,218 0.0058
09 May 2023 16:35:00 1531398 22268808 oifs_43r3_ps_0973_1997050100_123_966_12183617_0 4,665,600 27,515 0.0059
09 May 2023 16:34:06 1531398 22268808 oifs_43r3_ps_0973_1997050100_123_966_12183617_0 2,332,800 12,939 0.0055
09 May 2023 16:33:03 1531398 22268808 oifs_43r3_ps_0973_1997050100_123_966_12183617_0 864,000 4,719 0.0055


©2024 climateprediction.net