climateprediction.net home page
Task 22296101

Task 22296101

Name oifs_43r3_ps_0415_1993050100_123_962_12179059_1
Workunit 12179059
Created 6 Jan 2023, 18:41:35 UTC
Sent 20 Jan 2023, 2:11:14 UTC
Report deadline 21 Mar 2023, 2:11:14 UTC
Received 20 Jan 2023, 17:33:10 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1532184
Run time 13 hours 23 min 21 sec
CPU time 13 hours 20 min 32 sec
Validate state Invalid
Credit 8,467.20
Device peak FLOPS 6.86 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.50 MB
Peak swap size 4,974.80 MB
Peak disk usage 1,218.66 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
c-client/slots/15/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001668
  15:11:48 STEP 1669 H=1669:00 +CPU= 21.973
  15:12:09 STEP 1670 H=1670:00 +CPU= 21.909
  15:12:42 STEP 1671 H=1671:00 +CPU= 33.005
  15:13:05 STEP 1672 H=1672:00 +CPU= 22.555
  15:13:27 STEP 1673 H=1673:00 +CPU= 21.607
  15:14:00 STEP 1674 H=1674:00 +CPU= 33.603
  15:14:22 STEP 1675 H=1675:00 +CPU= 21.795
  15:14:44 STEP 1676 H=1676:00 +CPU= 22.021
  15:15:17 STEP 1677 H=1677:00 +CPU= 32.934
  15:15:39 STEP 1678 H=1678:00 +CPU= 22.208
  15:16:02 STEP 1679 H=1679:00 +CPU= 22.279
  15:16:40 STEP 1680 H=1680:00 +CPU= 38.408
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0415_1993050100_123_962_12179059_1_r2053049523_69.zip
Uploading the intermediate file: upload_file_69.zip
  15:17:01 STEP 1681 H=1681:00 +CPU= 21.271
Uploading trickle at timestep: 6048000
  15:17:23 STEP 1682 H=1682:00 +CPU= 21.551
  15:17:55 STEP 1683 H=1683:00 +CPU= 32.500
  15:18:18 STEP 1684 H=1684:00 +CPU= 22.083
  15:18:39 STEP 1685 H=1685:00 +CPU= 21.366
  15:19:12 STEP 1686 H=1686:00 +CPU= 33.501
  15:19:34 STEP 1687 H=1687:00 +CPU= 21.831
  15:19:56 STEP 1688 H=1688:00 +CPU= 22.154
  15:20:30 STEP 1689 H=1689:00 +CPU= 33.275
  15:20:52 STEP 1690 H=1690:00 +CPU= 22.489
  15:21:14 STEP 1691 H=1691:00 +CPU= 21.665
  15:21:50 STEP 1692 H=1692:00 +CPU= 35.854
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001692
  15:22:11 STEP 1693 H=1693:00 +CPU= 21.557
  15:22:33 STEP 1694 H=1694:00 +CPU= 21.888
  15:23:06 STEP 1695 H=1695:00 +CPU= 32.879
  15:23:29 STEP 1696 H=1696:00 +CPU= 22.421
  15:23:50 STEP 1697 H=1697:00 +CPU= 21.727
  15:24:25 STEP 1698 H=1698:00 +CPU= 34.308
  15:24:46 STEP 1699 H=1699:00 +CPU= 21.610
  15:25:08 STEP 1700 H=1700:00 +CPU= 22.075
  15:25:42 STEP 1701 H=1701:00 +CPU= 33.662
  15:26:05 STEP 1702 H=1702:00 +CPU= 22.578
  15:26:27 STEP 1703 H=1703:00 +CPU= 22.127
  15:27:05 STEP 1704 H=1704:00 +CPU= 38.398
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0415_1993050100_123_962_12179059_1_r2053049523_70.zip
Uploading the intermediate file: upload_file_70.zip
  15:27:26 STEP 1705 H=1705:00 +CPU= 21.092
  15:27:48 STEP 1706 H=1706:00 +CPU= 21.524
  15:28:21 STEP 1707 H=1707:00 +CPU= 32.861
  15:28:43 STEP 1708 H=1708:00 +CPU= 21.977
  15:29:04 STEP 1709 H=1709:00 +CPU= 21.363
  15:29:38 STEP 1710 H=1710:00 +CPU= 33.592
  15:30:00 STEP 1711 H=1711:00 +CPU= 21.733
  15:30:22 STEP 1712 H=1712:00 +CPU= 22.082
  15:30:55 STEP 1713 H=1713:00 +CPU= 33.698
  15:31:18 STEP 1714 H=1714:00 +CPU= 22.472
  15:31:40 STEP 1715 H=1715:00 +CPU= 21.831
  15:32:16 STEP 1716 H=1716:00 +CPU= 35.861
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001716
  15:32:37 STEP 1717 H=1717:00 +CPU= 21.449
  15:32:59 STEP 1718 H=1718:00 +CPU= 21.796
  15:33:32 STEP 1719 H=1719:00 +CPU= 32.915
  15:33:54 STEP 1720 H=1720:00 +CPU= 22.348
  15:34:16 STEP 1721 H=1721:00 +CPU= 21.437
  15:34:49 STEP 1722 H=1722:00 +CPU= 33.706
  15:35:11 STEP 1723 H=1723:00 +CPU= 22.074
  15:35:33 STEP 1724 H=1724:00 +CPU= 22.053
  15:36:07 STEP 1725 H=1725:00 +CPU= 33.192
  15:36:29 STEP 1726 H=1726:00 +CPU= 22.331
  15:36:51 STEP 1727 H=1727:00 +CPU= 22.170
  15:37:29 STEP 1728 H=1728:00 +CPU= 38.273
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0415_1993050100_123_962_12179059_1_r2053049523_71.zip
Uploading the intermediate file: upload_file_71.zip
  15:37:51 STEP 1729 H=1729:00 +CPU= 21.139
  15:38:12 STEP 1730 H=1730:00 +CPU= 21.519
  15:38:45 STEP 1731 H=1731:00 +CPU= 32.694
  15:39:07 STEP 1732 H=1732:00 +CPU= 22.132
  15:39:28 STEP 1733 H=1733:00 +CPU= 21.380
  15:40:02 STEP 1734 H=1734:00 +CPU= 33.669
  15:40:24 STEP 1735 H=1735:00 +CPU= 21.751
  15:40:46 STEP 1736 H=1736:00 +CPU= 22.058
  15:41:19 STEP 1737 H=1737:00 +CPU= 33.264
  15:41:42 STEP 1738 H=1738:00 +CPU= 22.409
  15:42:03 STEP 1739 H=1739:00 +CPU= 21.502
  15:42:39 STEP 1740 H=1740:00 +CPU= 36.319
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001740
  15:43:01 STEP 1741 H=1741:00 +CPU= 21.770
  15:43:23 STEP 1742 H=1742:00 +CPU= 22.045
  15:43:57 STEP 1743 H=1743:00 +CPU= 33.921
  15:44:20 STEP 1744 H=1744:00 +CPU= 22.309
  15:44:41 STEP 1745 H=1745:00 +CPU= 21.416
  15:45:15 STEP 1746 H=1746:00 +CPU= 33.778
  15:45:37 STEP 1747 H=1747:00 +CPU= 22.165
  15:46:00 STEP 1748 H=1748:00 +CPU= 22.595
  15:46:34 STEP 1749 H=1749:00 +CPU= 33.887
  15:46:56 STEP 1750 H=1750:00 +CPU= 22.476
  15:47:19 STEP 1751 H=1751:00 +CPU= 22.532
  15:47:58 STEP 1752 H=1752:00 +CPU= 38.940
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0415_1993050100_123_962_12179059_1_r2053049523_72.zip
Uploading the intermediate file: upload_file_72.zip
  15:48:19 STEP 1753 H=1753:00 +CPU= 21.317
  15:48:41 STEP 1754 H=1754:00 +CPU= 21.775
  15:49:15 STEP 1755 H=1755:00 +CPU= 33.387
  15:49:38 STEP 1756 H=1756:00 +CPU= 23.035
  15:50:00 STEP 1757 H=1757:00 +CPU= 22.244
  15:50:35 STEP 1758 H=1758:00 +CPU= 34.984
  15:50:58 STEP 1759 H=1759:00 +CPU= 22.736
  15:51:21 STEP 1760 H=1760:00 +CPU= 23.050
  15:51:55 STEP 1761 H=1761:00 +CPU= 34.274
  15:52:18 STEP 1762 H=1762:00 +CPU= 23.108
  15:52:41 STEP 1763 H=1763:00 +CPU= 22.243
  15:53:18 STEP 1764 H=1764:00 +CPU= 37.565
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001764
  15:53:40 STEP 1765 H=1765:00 +CPU= 21.792
  15:54:03 STEP 1766 H=1766:00 +CPU= 22.495
  15:54:37 STEP 1767 H=1767:00 +CPU= 34.301
  15:54:59 STEP 1768 H=1768:00 +CPU= 22.559
  15:55:21 STEP 1769 H=1769:00 +CPU= 21.996
  15:55:57 STEP 1770 H=1770:00 +CPU= 35.201
  15:56:19 STEP 1771 H=1771:00 +CPU= 22.376
  15:56:42 STEP 1772 H=1772:00 +CPU= 22.756
  15:57:16 STEP 1773 H=1773:00 +CPU= 34.245
  15:57:39 STEP 1774 H=1774:00 +CPU= 23.209
  15:58:02 STEP 1775 H=1775:00 +CPU= 22.917
  15:58:43 STEP 1776 H=1776:00 +CPU= 40.329
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0415_1993050100_123_962_12179059_1_r2053049523_73.zip
Uploading the intermediate file: upload_file_73.zip
  15:59:04 STEP 1777 H=1777:00 +CPU= 21.790
  15:59:27 STEP 1778 H=1778:00 +CPU= 22.468
  16:00:01 STEP 1779 H=1779:00 +CPU= 33.795
  16:00:24 STEP 1780 H=1780:00 +CPU= 22.824
  16:00:46 STEP 1781 H=1781:00 +CPU= 22.037
  16:01:21 STEP 1782 H=1782:00 +CPU= 34.881
  16:01:43 STEP 1783 H=1783:00 +CPU= 22.574
  16:02:06 STEP 1784 H=1784:00 +CPU= 22.720
  16:02:40 STEP 1785 H=1785:00 +CPU= 34.587
  16:03:04 STEP 1786 H=1786:00 +CPU= 23.152
  16:03:26 STEP 1787 H=1787:00 +CPU= 22.206
  16:04:03 STEP 1788 H=1788:00 +CPU= 37.470
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001788
  16:04:25 STEP 1789 H=1789:00 +CPU= 21.926
  16:04:47 STEP 1790 H=1790:00 +CPU= 22.070
  16:05:22 STEP 1791 H=1791:00 +CPU= 34.345
  16:05:44 STEP 1792 H=1792:00 +CPU= 22.547
  16:06:06 STEP 1793 H=1793:00 +CPU= 21.741
  16:06:41 STEP 1794 H=1794:00 +CPU= 34.983
  16:07:03 STEP 1795 H=1795:00 +CPU= 22.336
  16:07:26 STEP 1796 H=1796:00 +CPU= 22.731
  16:08:01 STEP 1797 H=1797:00 +CPU= 34.365
  16:08:24 STEP 1798 H=1798:00 +CPU= 23.390
  16:08:47 STEP 1799 H=1799:00 +CPU= 23.003
  16:09:27 STEP 1800 H=1800:00 +CPU= 39.899
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0415_1993050100_123_962_12179059_1_r2053049523_74.zip
Uploading the intermediate file: upload_file_74.zip
  16:09:49 STEP 1801 H=1801:00 +CPU= 21.913
  16:10:11 STEP 1802 H=1802:00 +CPU= 22.131
  16:10:45 STEP 1803 H=1803:00 +CPU= 34.209
  16:11:08 STEP 1804 H=1804:00 +CPU= 22.929
  16:11:30 STEP 1805 H=1805:00 +CPU= 21.999
  16:12:05 STEP 1806 H=1806:00 +CPU= 34.767
  16:12:28 STEP 1807 H=1807:00 +CPU= 22.575
  16:12:50 STEP 1808 H=1808:00 +CPU= 22.831
  16:13:25 STEP 1809 H=1809:00 +CPU= 34.233
  16:13:48 STEP 1810 H=1810:00 +CPU= 23.250
  16:14:10 STEP 1811 H=1811:00 +CPU= 22.290
  16:14:48 STEP 1812 H=1812:00 +CPU= 37.763
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001812
  16:15:10 STEP 1813 H=1813:00 +CPU= 22.405
  16:15:33 STEP 1814 H=1814:00 +CPU= 22.491
  16:16:08 STEP 1815 H=1815:00 +CPU= 35.109
  16:16:32 STEP 1816 H=1816:00 +CPU= 23.659
  16:16:55 STEP 1817 H=1817:00 +CPU= 23.071
  16:17:31 STEP 1818 H=1818:00 +CPU= 36.468
  16:17:55 STEP 1819 H=1819:00 +CPU= 23.621
  16:18:15 STEP 1820 H=1820:00 +CPU= 19.725
  16:18:41 STEP 1821 H=1821:00 +CPU= 26.275
  16:18:58 STEP 1822 H=1822:00 +CPU= 16.843
  16:19:15 STEP 1823 H=1823:00 +CPU= 16.831
  16:19:46 STEP 1824 H=1824:00 +CPU= 31.431
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0415_1993050100_123_962_12179059_1_r2053049523_75.zip
Uploading the intermediate file: upload_file_75.zip
  16:20:03 STEP 1825 H=1825:00 +CPU= 16.618
  16:20:20 STEP 1826 H=1826:00 +CPU= 16.898
  16:20:46 STEP 1827 H=1827:00 +CPU= 26.318
  16:21:04 STEP 1828 H=1828:00 +CPU= 17.865
  16:21:22 STEP 1829 H=1829:00 +CPU= 18.520
  16:21:56 STEP 1830 H=1830:00 +CPU= 33.974
  16:22:18 STEP 1831 H=1831:00 +CPU= 21.977
  16:22:41 STEP 1832 H=1832:00 +CPU= 22.416
  16:23:14 STEP 1833 H=1833:00 +CPU= 33.496
  16:23:32 STEP 1834 H=1834:00 +CPU= 17.988
  16:23:49 STEP 1835 H=1835:00 +CPU= 16.793
  16:24:18 STEP 1836 H=1836:00 +CPU= 28.788
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001836
  16:24:36 STEP 1837 H=1837:00 +CPU= 18.049
  16:24:53 STEP 1838 H=1838:00 +CPU= 16.960
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4150MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4150 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162456:1674228296:47353.165] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-INPUT LOOP1  (1205) 
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162457:1674228297:47354.167] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:20455:20455] [20230120:162457:1674228297:47354.167] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001292200  radintg_                  830  radintg.F90
oifs_43r3_model.e  0000000001270124  raddrv_                   537  raddrv.F90
oifs_43r3_model.e  0000000000BFD839  gp_model_                 482  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0415
(argv4) batchid: 962
(argv5) wuid: 12179059
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/15
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d6f8072b2448e04b9c234d029ffd44f6 to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0415_1993050100_123_962_12179059.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0415_1993050100_123_962_12179059.zip
ic_ancil_file: ic_ancil_12179059
ifsdata_file: ifsdata_12179059
climate_data_file: clim_data_12179059
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_47530acef7dd72cefe43e0ce09a1ba92 to: /var/lib/boinc-client/slots/15/ic_ancil_12179059.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/15/ic_ancil_12179059.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9fe690bf2fcdac049ceff82f61948fab to: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12179059.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12179059.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/15/159l_2/clim_data_12179059.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/15/159l_2/clim_data_12179059.zip
Checking for progress XML file: /var/lib/boinc-client/slots/15/progress_file_12179059.xml
Opened progress file ok : /var/lib/boinc-client/slots/15/progress_file_12179059.xml
-- Model is restarting --
Adjusting last_iter, 1839, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/15/progress_file_12179059.xml
last_cpu_time: 47324
upload_file_number: 76
last_iter: 1823
last_upload: 6566400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0415_1993050100_123_962_12179059_1_r2053049523
The child process has been launched with process id: 1462
Executing the command: /var/lib/boinc-client/slots/15/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171059:1674231059:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:12:00 STEP 1824 H=1824:00 +CPU= 44.648
  17:12:26 STEP 1825 H=1825:00 +CPU= 25.045
  17:12:47 STEP 1826 H=1826:00 +CPU= 21.724
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4032MB (heap), 4403MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4032 MB (maxheap), 4403 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-RADLSW       (1210) 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               RADIATION_SCHEME 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                radiation_interface:radiation 
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171253:1674231173:114.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 radiation_ifs_rrtm:gas_optics 
double free or corruption (top)
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171255:1674231175:115.954] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:W31Ub20:1:1:1462:1462] [20230120:171255:1674231175:115.954] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000018BC7AE  radiation_ifs_rrt         479  radiation_ifs_rrtm.F90
oifs_43r3_model.e  0000000001862293  radiation_interfa         311  radiation_interface.F90
oifs_43r3_model.e  00000000012D30E5  radiation_scheme_         484  radiation_scheme.F90
oifs_43r3_model.e  000000000129A415  radintg_                 1455  radintg.F90
oifs_43r3_model.e  0000000001270124  raddrv_                   537  raddrv.F90
oifs_43r3_model.e  0000000000BFD839  gp_model_                 482  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0415
(argv4) batchid: 962
(argv5) wuid: 12179059
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/15
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d6f8072b2448e04b9c234d029ffd44f6 to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0415_1993050100_123_962_12179059.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0415_1993050100_123_962_12179059.zip
ic_ancil_file: ic_ancil_12179059
ifsdata_file: ifsdata_12179059
climate_data_file: clim_data_12179059
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_47530acef7dd72cefe43e0ce09a1ba92 to: /var/lib/boinc-client/slots/15/ic_ancil_12179059.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/15/ic_ancil_12179059.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9fe690bf2fcdac049ceff82f61948fab to: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12179059.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12179059.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/15/159l_2/clim_data_12179059.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/15/159l_2/clim_data_12179059.zip
Checking for progress XML file: /var/lib/boinc-client/slots/15/progress_file_12179059.xml
Opened progress file ok : /var/lib/boinc-client/slots/15/progress_file_12179059.xml
-- Model is restarting --
Adjusting last_iter, 1827, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/15/progress_file_12179059.xml
last_cpu_time: 47421
upload_file_number: 76
last_iter: 1823
last_upload: 6566400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0415_1993050100_123_962_12179059_1_r2053049523
The child process has been launched with process id: 1451
Executing the command: /var/lib/boinc-client/slots/15/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1451:1451] [20230120:181219:1674234739:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:13:18 STEP 1824 H=1824:00 +CPU= 42.587
  18:13:38 STEP 1825 H=1825:00 +CPU= 20.302
  18:14:00 STEP 1826 H=1826:00 +CPU= 21.979
  18:14:32 STEP 1827 H=1827:00 +CPU= 32.142
  18:14:52 STEP 1828 H=1828:00 +CPU= 20.137
  18:15:13 STEP 1829 H=1829:00 +CPU= 21.028
  18:15:46 STEP 1830 H=1830:00 +CPU= 32.299
  18:16:06 STEP 1831 H=1831:00 +CPU= 20.564
  18:16:28 STEP 1832 H=1832:00 +CPU= 21.715
  18:17:00 STEP 1833 H=1833:00 +CPU= 31.895
  18:17:21 STEP 1834 H=1834:00 +CPU= 20.828
  18:17:42 STEP 1835 H=1835:00 +CPU= 21.082
  18:18:16 STEP 1836 H=1836:00 +CPU= 34.641
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+001836
  18:18:37 STEP 1837 H=1837:00 +CPU= 20.628
  18:18:58 STEP 1838 H=1838:00 +CPU= 21.125
  18:19:30 STEP 1839 H=1839:00 +CPU= 31.653
  18:19:51 STEP 1840 H=1840:00 +CPU= 20.857
  18:20:12 STEP 1841 H=1841:00 +CPU= 21.457
  18:20:45 STEP 1842 H=1842:00 +CPU= 32.526
  18:21:05 STEP 1843 H=1843:00 +CPU= 20.312
  18:21:26 STEP 1844 H=1844:00 +CPU= 21.056
  18:21:57 STEP 1845 H=1845:00 +CPU= 30.904
  18:22:17 STEP 1846 H=1846:00 +CPU= 19.805
  18:22:38 STEP 1847 H=1847:00 +CPU= 20.702
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0415
(argv4) batchid: 962
(argv5) wuid: 12179059
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/15
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179059
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d6f8072b2448e04b9c234d029ffd44f6 to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0415_1993050100_123_962_12179059.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0415_1993050100_123_962_12179059.zip
ic_ancil_file: ic_ancil_12179059
ifsdata_file: ifsdata_12179059
climate_data_file: clim_data_12179059
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_47530acef7dd72cefe43e0ce09a1ba92 to: /var/lib/boinc-client/slots/15/ic_ancil_12179059.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/15/ic_ancil_12179059.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9fe690bf2fcdac049ceff82f61948fab to: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12179059.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12179059.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/15/159l_2/clim_data_12179059.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/15/159l_2/clim_data_12179059.zip
Checking for progress XML file: /var/lib/boinc-client/slots/15/progress_file_12179059.xml
Opened progress file ok : /var/lib/boinc-client/slots/15/progress_file_12179059.xml
-- Model is restarting --
Adjusting last_iter, 1848, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/15/progress_file_12179059.xml
last_cpu_time: 48031
upload_file_number: 76
last_iter: 1847
last_upload: 6566400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0415_1993050100_123_962_12179059_1_r2053049523
The child process has been launched with process id: 1411
Executing the command: /var/lib/boinc-client/slots/15/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182917:1674235757:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
ECCODES ERROR   :  IO ERROR: Inappropriate ioctl for device (Inappropriate ioctl for device)
 GRIB_READ_BYTES_REAL8        50000  FAILED         -11
 GRIB_API ERROR MSG: Input output problem
MPL_ABORT: CALLED FROM PROCESSOR      1 THRD     1
 MPL_ABORT: THRD           1   GRIB_READ_BYTES_REAL8 FAILED
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182922:1674235762:4.072] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1348 MB (maxheap), 703 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182922:1674235762:4.072] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182922:1674235762:4.072] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182922:1674235762:4.072] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182922:1674235762:4.072] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182922:1674235762:4.072] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182922:1674235762:4.072] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       RERESF 
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182922:1674235762:4.072] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        IOSTREAM_MIX:IO_GET 
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182922:1674235762:4.072] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         IOSTREAM_MIX:READ_RECORD 
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182922:1674235762:4.072] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          >MPL-IOSTREAM READ_RECORD ( 650) 
[EC_DRHOOK:W31Ub20:1:1:1411:1411] [20230120:182922:1674235762:4.072] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GRIB_API:IGRIB_READ_BYTES_REAL8 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/15/oifs_43r3_model.exe' : sigcontextptr=0x7ffd09d1f400
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 16) with plain hex-dump
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [0]: [0x839273] : 0x839273
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [1]: [0x839879] : 0x839879
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [2]: [0x8c2957] : 0x8c2957
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [3]: [0x8c3229] : 0x8c3229
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [4]: [0xa048b9] : 0xa048b9
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [5]: [0x9e3c9a] : 0x9e3c9a
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [6]: [0x9db20d] : 0x9db20d
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [7]: [0x54cfed] : 0x54cfed
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [8]: [0x546a18] : 0x546a18
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [9]: [0x546412] : 0x546412
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [10]: [0x545e28] : 0x545e28
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [11]: [0x40708d] : 0x40708d
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [12]: [0x40220f] : 0x40220f
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [13]: [0x4021a2] : 0x4021a2
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [14]: [0x1dca390] : 0x1dca390
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [15]: [0x40206e] : 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 SDL_TRACEBACK: Done INTEL_TRBK, THRD =            1
abort: 
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:31:00 1532184 22296101 oifs_43r3_ps_0415_1993050100_123_962_12179059_1 6,048,000 43,236 0.0071
09 May 2023 16:30:55 1532184 22296101 oifs_43r3_ps_0415_1993050100_123_962_12179059_1 5,184,000 36,988 0.0071
09 May 2023 16:30:43 1532184 22296101 oifs_43r3_ps_0415_1993050100_123_962_12179059_1 4,320,000 30,739 0.0071
09 May 2023 16:30:31 1532184 22296101 oifs_43r3_ps_0415_1993050100_123_962_12179059_1 3,456,000 24,481 0.0071
09 May 2023 16:30:25 1532184 22296101 oifs_43r3_ps_0415_1993050100_123_962_12179059_1 2,592,000 18,232 0.0070
09 May 2023 16:30:13 1532184 22296101 oifs_43r3_ps_0415_1993050100_123_962_12179059_1 1,728,000 12,063 0.0070
09 May 2023 16:30:01 1532184 22296101 oifs_43r3_ps_0415_1993050100_123_962_12179059_1 864,000 5,885 0.0068


©2024 climateprediction.net