climateprediction.net home page
Task 22306913

Task 22306913

Name oifs_43r3_ps_2530_2021050100_123_990_12206764_0
Workunit 12206764
Created 9 Feb 2023, 15:22:50 UTC
Sent 9 Feb 2023, 15:26:19 UTC
Report deadline 10 Apr 2023, 15:26:19 UTC
Received 10 Feb 2023, 5:01:11 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1536819
Run time 12 hours 46 min 46 sec
CPU time 12 hours 39 min 53 sec
Validate state Invalid
Credit 10,644.48
Device peak FLOPS 5.13 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.09
x86_64-pc-linux-gnu
Peak working set size 4,800.29 MB
Peak swap size 4,978.14 MB
Peak disk usage 1,210.07 MB
Stderr
<core_client_version>7.22.0</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
44:00 +CPU= 30.257
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001644
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001644
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001644
  19:10:09 STEP 1645 H=1645:00 +CPU= 17.401
  19:10:27 STEP 1646 H=1646:00 +CPU= 17.897
  19:10:54 STEP 1647 H=1647:00 +CPU= 26.880
  19:11:13 STEP 1648 H=1648:00 +CPU= 17.775
  19:11:30 STEP 1649 H=1649:00 +CPU= 17.215
  19:11:58 STEP 1650 H=1650:00 +CPU= 27.522
  19:12:15 STEP 1651 H=1651:00 +CPU= 17.192
  19:12:33 STEP 1652 H=1652:00 +CPU= 17.728
  19:13:00 STEP 1653 H=1653:00 +CPU= 26.841
  19:13:17 STEP 1654 H=1654:00 +CPU= 17.669
  19:13:35 STEP 1655 H=1655:00 +CPU= 17.705
  19:14:07 STEP 1656 H=1656:00 +CPU= 31.803
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001656
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001656
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001644
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_68.zip
Uploading the intermediate file: upload_file_68.zip
  19:14:25 STEP 1657 H=1657:00 +CPU= 17.218
  19:14:43 STEP 1658 H=1658:00 +CPU= 17.905
  19:15:10 STEP 1659 H=1659:00 +CPU= 26.813
  19:15:27 STEP 1660 H=1660:00 +CPU= 17.734
  19:15:45 STEP 1661 H=1661:00 +CPU= 17.117
  19:16:12 STEP 1662 H=1662:00 +CPU= 27.460
  19:16:29 STEP 1663 H=1663:00 +CPU= 17.143
  19:16:47 STEP 1664 H=1664:00 +CPU= 17.698
  19:17:14 STEP 1665 H=1665:00 +CPU= 26.831
  19:17:32 STEP 1666 H=1666:00 +CPU= 17.824
  19:17:50 STEP 1667 H=1667:00 +CPU= 17.192
  19:18:20 STEP 1668 H=1668:00 +CPU= 30.161
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001668
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001668
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001668
  19:18:37 STEP 1669 H=1669:00 +CPU= 17.326
  19:18:55 STEP 1670 H=1670:00 +CPU= 17.889
  19:19:22 STEP 1671 H=1671:00 +CPU= 26.864
  19:19:41 STEP 1672 H=1672:00 +CPU= 17.797
  19:19:58 STEP 1673 H=1673:00 +CPU= 17.239
  19:20:26 STEP 1674 H=1674:00 +CPU= 27.592
  19:20:43 STEP 1675 H=1675:00 +CPU= 17.210
  19:21:01 STEP 1676 H=1676:00 +CPU= 17.746
  19:21:28 STEP 1677 H=1677:00 +CPU= 26.866
  19:21:45 STEP 1678 H=1678:00 +CPU= 17.665
  19:22:03 STEP 1679 H=1679:00 +CPU= 17.735
  19:22:35 STEP 1680 H=1680:00 +CPU= 31.949
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001680
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001680
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001668
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_69.zip
Uploading the intermediate file: upload_file_69.zip
  19:22:53 STEP 1681 H=1681:00 +CPU= 17.192
Uploading trickle at timestep: 6048000
  19:23:11 STEP 1682 H=1682:00 +CPU= 17.766
  19:23:38 STEP 1683 H=1683:00 +CPU= 26.876
  19:23:56 STEP 1684 H=1684:00 +CPU= 17.749
  19:24:13 STEP 1685 H=1685:00 +CPU= 17.195
  19:24:40 STEP 1686 H=1686:00 +CPU= 27.470
  19:24:58 STEP 1687 H=1687:00 +CPU= 17.215
  19:25:16 STEP 1688 H=1688:00 +CPU= 17.659
  19:25:43 STEP 1689 H=1689:00 +CPU= 26.981
  19:26:01 STEP 1690 H=1690:00 +CPU= 17.714
  19:26:18 STEP 1691 H=1691:00 +CPU= 17.221
  19:26:49 STEP 1692 H=1692:00 +CPU= 30.179
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001692
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001692
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001692
  19:27:06 STEP 1693 H=1693:00 +CPU= 17.218
  19:27:24 STEP 1694 H=1694:00 +CPU= 17.790
  19:27:51 STEP 1695 H=1695:00 +CPU= 26.825
  19:28:09 STEP 1696 H=1696:00 +CPU= 17.758
  19:28:26 STEP 1697 H=1697:00 +CPU= 17.205
  19:28:54 STEP 1698 H=1698:00 +CPU= 27.575
  19:29:11 STEP 1699 H=1699:00 +CPU= 17.143
  19:29:29 STEP 1700 H=1700:00 +CPU= 17.700
  19:29:56 STEP 1701 H=1701:00 +CPU= 26.861
  19:30:14 STEP 1702 H=1702:00 +CPU= 17.761
  19:30:32 STEP 1703 H=1703:00 +CPU= 17.717
  19:31:04 STEP 1704 H=1704:00 +CPU= 31.878
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001704
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001704
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001692
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_70.zip
Uploading the intermediate file: upload_file_70.zip
  19:31:22 STEP 1705 H=1705:00 +CPU= 17.151
  19:31:40 STEP 1706 H=1706:00 +CPU= 17.717
  19:32:07 STEP 1707 H=1707:00 +CPU= 26.857
  19:32:24 STEP 1708 H=1708:00 +CPU= 17.723
  19:32:42 STEP 1709 H=1709:00 +CPU= 17.206
  19:33:09 STEP 1710 H=1710:00 +CPU= 27.493
  19:33:27 STEP 1711 H=1711:00 +CPU= 17.198
  19:33:44 STEP 1712 H=1712:00 +CPU= 17.660
  19:34:12 STEP 1713 H=1713:00 +CPU= 27.044
  19:34:30 STEP 1714 H=1714:00 +CPU= 17.852
  19:34:47 STEP 1715 H=1715:00 +CPU= 17.310
  19:35:17 STEP 1716 H=1716:00 +CPU= 30.054
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001716
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001716
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001716
  19:35:35 STEP 1717 H=1717:00 +CPU= 17.131
  19:35:52 STEP 1718 H=1718:00 +CPU= 17.692
  19:36:19 STEP 1719 H=1719:00 +CPU= 26.720
  19:36:37 STEP 1720 H=1720:00 +CPU= 17.710
  19:36:55 STEP 1721 H=1721:00 +CPU= 17.161
  19:37:22 STEP 1722 H=1722:00 +CPU= 27.434
  19:37:39 STEP 1723 H=1723:00 +CPU= 17.116
  19:37:57 STEP 1724 H=1724:00 +CPU= 17.658
  19:38:24 STEP 1725 H=1725:00 +CPU= 26.901
  19:38:42 STEP 1726 H=1726:00 +CPU= 17.787
  19:39:00 STEP 1727 H=1727:00 +CPU= 17.786
  19:39:32 STEP 1728 H=1728:00 +CPU= 31.774
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001728
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001728
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001716
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_71.zip
Uploading the intermediate file: upload_file_71.zip
  19:39:49 STEP 1729 H=1729:00 +CPU= 17.125
  19:40:07 STEP 1730 H=1730:00 +CPU= 17.670
  19:40:34 STEP 1731 H=1731:00 +CPU= 26.891
  19:40:52 STEP 1732 H=1732:00 +CPU= 17.771
  19:41:09 STEP 1733 H=1733:00 +CPU= 17.236
  19:41:37 STEP 1734 H=1734:00 +CPU= 27.499
  19:41:54 STEP 1735 H=1735:00 +CPU= 17.231
  19:42:12 STEP 1736 H=1736:00 +CPU= 17.704
  19:42:39 STEP 1737 H=1737:00 +CPU= 27.055
  19:42:57 STEP 1738 H=1738:00 +CPU= 17.836
  19:43:15 STEP 1739 H=1739:00 +CPU= 17.380
  19:43:45 STEP 1740 H=1740:00 +CPU= 30.031
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001740
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001740
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001740
  19:44:02 STEP 1741 H=1741:00 +CPU= 17.148
  19:44:20 STEP 1742 H=1742:00 +CPU= 17.703
  19:44:47 STEP 1743 H=1743:00 +CPU= 26.751
  19:45:05 STEP 1744 H=1744:00 +CPU= 17.806
  19:45:22 STEP 1745 H=1745:00 +CPU= 17.166
  19:45:50 STEP 1746 H=1746:00 +CPU= 27.492
  19:46:07 STEP 1747 H=1747:00 +CPU= 17.149
  19:46:25 STEP 1748 H=1748:00 +CPU= 17.750
  19:46:52 STEP 1749 H=1749:00 +CPU= 26.821
  19:47:10 STEP 1750 H=1750:00 +CPU= 17.845
  19:47:28 STEP 1751 H=1751:00 +CPU= 17.799
  19:48:00 STEP 1752 H=1752:00 +CPU= 31.708
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001752
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001752
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001740
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_72.zip
Uploading the intermediate file: upload_file_72.zip
  19:48:17 STEP 1753 H=1753:00 +CPU= 17.104
  19:48:35 STEP 1754 H=1754:00 +CPU= 17.661
  19:49:02 STEP 1755 H=1755:00 +CPU= 27.005
  19:49:20 STEP 1756 H=1756:00 +CPU= 17.739
  19:49:37 STEP 1757 H=1757:00 +CPU= 17.216
  19:50:05 STEP 1758 H=1758:00 +CPU= 27.540
  19:50:22 STEP 1759 H=1759:00 +CPU= 17.244
  19:50:40 STEP 1760 H=1760:00 +CPU= 17.777
  19:51:07 STEP 1761 H=1761:00 +CPU= 26.959
  19:51:25 STEP 1762 H=1762:00 +CPU= 17.780
  19:51:43 STEP 1763 H=1763:00 +CPU= 17.193
  19:52:13 STEP 1764 H=1764:00 +CPU= 30.064
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001764
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001764
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001764
  19:52:30 STEP 1765 H=1765:00 +CPU= 17.188
  19:52:48 STEP 1766 H=1766:00 +CPU= 17.671
  19:53:15 STEP 1767 H=1767:00 +CPU= 26.859
  19:53:33 STEP 1768 H=1768:00 +CPU= 17.795
  19:53:50 STEP 1769 H=1769:00 +CPU= 17.176
  19:54:18 STEP 1770 H=1770:00 +CPU= 27.501
  19:54:35 STEP 1771 H=1771:00 +CPU= 17.159
  19:54:53 STEP 1772 H=1772:00 +CPU= 17.758
  19:55:20 STEP 1773 H=1773:00 +CPU= 26.775
  19:55:38 STEP 1774 H=1774:00 +CPU= 17.750
  19:55:55 STEP 1775 H=1775:00 +CPU= 17.715
  19:56:27 STEP 1776 H=1776:00 +CPU= 31.630
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001776
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001776
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001764
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_73.zip
Uploading the intermediate file: upload_file_73.zip
  19:56:45 STEP 1777 H=1777:00 +CPU= 17.118
  19:57:03 STEP 1778 H=1778:00 +CPU= 17.562
  19:57:30 STEP 1779 H=1779:00 +CPU= 26.967
  19:57:48 STEP 1780 H=1780:00 +CPU= 17.833
  19:58:05 STEP 1781 H=1781:00 +CPU= 17.180
  19:58:33 STEP 1782 H=1782:00 +CPU= 27.603
  19:58:50 STEP 1783 H=1783:00 +CPU= 17.267
  19:59:08 STEP 1784 H=1784:00 +CPU= 17.769
  19:59:35 STEP 1785 H=1785:00 +CPU= 26.927
  19:59:53 STEP 1786 H=1786:00 +CPU= 17.859
  20:00:10 STEP 1787 H=1787:00 +CPU= 17.290
  20:00:41 STEP 1788 H=1788:00 +CPU= 29.910
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001788
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001788
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001788
  20:00:58 STEP 1789 H=1789:00 +CPU= 17.154
  20:01:15 STEP 1790 H=1790:00 +CPU= 17.629
  20:01:42 STEP 1791 H=1791:00 +CPU= 26.805
  20:02:00 STEP 1792 H=1792:00 +CPU= 17.705
  20:02:18 STEP 1793 H=1793:00 +CPU= 17.124
  20:02:45 STEP 1794 H=1794:00 +CPU= 27.456
  20:03:03 STEP 1795 H=1795:00 +CPU= 17.114
  20:03:20 STEP 1796 H=1796:00 +CPU= 17.762
  20:03:47 STEP 1797 H=1797:00 +CPU= 26.714
  20:04:05 STEP 1798 H=1798:00 +CPU= 17.677
  20:04:23 STEP 1799 H=1799:00 +CPU= 17.716
  20:04:55 STEP 1800 H=1800:00 +CPU= 31.641
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001800
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001800
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001788
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_74.zip
Uploading the intermediate file: upload_file_74.zip
  20:05:12 STEP 1801 H=1801:00 +CPU= 17.082
  20:05:30 STEP 1802 H=1802:00 +CPU= 17.566
  20:05:57 STEP 1803 H=1803:00 +CPU= 26.985
  20:06:15 STEP 1804 H=1804:00 +CPU= 17.753
  20:06:32 STEP 1805 H=1805:00 +CPU= 17.179
  20:06:59 STEP 1806 H=1806:00 +CPU= 27.562
  20:07:17 STEP 1807 H=1807:00 +CPU= 17.262
  20:07:35 STEP 1808 H=1808:00 +CPU= 17.839
  20:08:02 STEP 1809 H=1809:00 +CPU= 26.918
  20:08:20 STEP 1810 H=1810:00 +CPU= 17.845
  20:08:37 STEP 1811 H=1811:00 +CPU= 17.275
  20:09:07 STEP 1812 H=1812:00 +CPU= 29.877
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001812
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001812
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001812
  20:09:25 STEP 1813 H=1813:00 +CPU= 17.126
  20:09:42 STEP 1814 H=1814:00 +CPU= 17.614
  20:10:09 STEP 1815 H=1815:00 +CPU= 26.789
  20:10:27 STEP 1816 H=1816:00 +CPU= 17.693
  20:10:44 STEP 1817 H=1817:00 +CPU= 17.089
  20:11:12 STEP 1818 H=1818:00 +CPU= 27.409
  20:11:29 STEP 1819 H=1819:00 +CPU= 17.120
  20:11:47 STEP 1820 H=1820:00 +CPU= 17.754
  20:12:14 STEP 1821 H=1821:00 +CPU= 26.733
  20:12:32 STEP 1822 H=1822:00 +CPU= 17.765
  20:12:50 STEP 1823 H=1823:00 +CPU= 17.757
  20:13:21 STEP 1824 H=1824:00 +CPU= 31.662
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001824
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001824
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_75.zip
Uploading the intermediate file: upload_file_75.zip
  20:13:39 STEP 1825 H=1825:00 +CPU= 17.109
  20:13:56 STEP 1826 H=1826:00 +CPU= 17.558
  20:14:23 STEP 1827 H=1827:00 +CPU= 26.936
  20:14:41 STEP 1828 H=1828:00 +CPU= 17.713
  20:14:58 STEP 1829 H=1829:00 +CPU= 17.137
  20:15:26 STEP 1830 H=1830:00 +CPU= 27.512
  20:15:43 STEP 1831 H=1831:00 +CPU= 17.239
  20:16:02 STEP 1832 H=1832:00 +CPU= 17.752
  20:16:29 STEP 1833 H=1833:00 +CPU= 26.900
  20:16:47 STEP 1834 H=1834:00 +CPU= 17.766
  20:17:04 STEP 1835 H=1835:00 +CPU= 17.254
  20:17:34 STEP 1836 H=1836:00 +CPU= 29.885
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001836
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001836
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001836
  20:17:52 STEP 1837 H=1837:00 +CPU= 17.175
  20:18:09 STEP 1838 H=1838:00 +CPU= 17.647
  20:18:36 STEP 1839 H=1839:00 +CPU= 26.782
  20:18:54 STEP 1840 H=1840:00 +CPU= 17.745
  20:19:12 STEP 1841 H=1841:00 +CPU= 17.147
  20:19:39 STEP 1842 H=1842:00 +CPU= 27.425
  20:19:56 STEP 1843 H=1843:00 +CPU= 17.097
  20:20:14 STEP 1844 H=1844:00 +CPU= 17.778
  20:20:41 STEP 1845 H=1845:00 +CPU= 26.769
  20:20:59 STEP 1846 H=1846:00 +CPU= 17.794
  20:21:17 STEP 1847 H=1847:00 +CPU= 17.741
  20:21:49 STEP 1848 H=1848:00 +CPU= 31.634
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001848
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001848
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001836
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_76.zip
Uploading the intermediate file: upload_file_76.zip
  20:22:06 STEP 1849 H=1849:00 +CPU= 17.113
  20:22:24 STEP 1850 H=1850:00 +CPU= 17.573
  20:22:51 STEP 1851 H=1851:00 +CPU= 26.955
  20:23:09 STEP 1852 H=1852:00 +CPU= 17.770
  20:23:26 STEP 1853 H=1853:00 +CPU= 17.147
  20:23:54 STEP 1854 H=1854:00 +CPU= 27.520
  20:24:11 STEP 1855 H=1855:00 +CPU= 17.231
  20:24:29 STEP 1856 H=1856:00 +CPU= 17.804
  20:24:56 STEP 1857 H=1857:00 +CPU= 26.852
  20:25:14 STEP 1858 H=1858:00 +CPU= 17.757
  20:25:31 STEP 1859 H=1859:00 +CPU= 17.230
  20:26:01 STEP 1860 H=1860:00 +CPU= 29.907
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001860
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001860
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001860
  20:26:18 STEP 1861 H=1861:00 +CPU= 17.172
  20:26:36 STEP 1862 H=1862:00 +CPU= 17.696
  20:27:03 STEP 1863 H=1863:00 +CPU= 26.818
  20:27:21 STEP 1864 H=1864:00 +CPU= 17.753
  20:27:39 STEP 1865 H=1865:00 +CPU= 17.163
  20:28:06 STEP 1866 H=1866:00 +CPU= 27.447
  20:28:23 STEP 1867 H=1867:00 +CPU= 17.152
  20:28:41 STEP 1868 H=1868:00 +CPU= 17.754
  20:29:08 STEP 1869 H=1869:00 +CPU= 26.741
  20:29:26 STEP 1870 H=1870:00 +CPU= 17.778
  20:29:44 STEP 1871 H=1871:00 +CPU= 17.745
  20:30:16 STEP 1872 H=1872:00 +CPU= 31.689
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001872
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001872
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001860
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_77.zip
Uploading the intermediate file: upload_file_77.zip
  20:30:33 STEP 1873 H=1873:00 +CPU= 17.145
  20:30:51 STEP 1874 H=1874:00 +CPU= 17.586
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2544MB (heap), 4800MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x815130 (old at (nil))
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2594 MB (maxheap), 4800 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SPCM 
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203116:1675996276:39877.025] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          TRSTOM 
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203117:1675996277:39878.027] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:science:1:1:62936:62936] [20230209:203117:1675996277:39878.028] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8cf0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE92AB  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000815D61  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC91D0  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000BABB05  trstom_                   360  trstom.F90
oifs_43r3_model.e  0000000000B779A3  spcm_                     287  spcm.F90
oifs_43r3_model.e  00000000004F8835  stepo_                    405  stepo.F90
oifs_43r3_model.e  00000000004B5C4C  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  000000000049EF09  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  000000000049E152  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049E048  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA4D0  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.09_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2530
(argv4) batchid: 990
(argv5) wuid: 12206764
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/4
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.09
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.09_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/4/oifs_43r3_ps_app_1.09_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/4/oifs_43r3_ps_app_1.09_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9dd820021499ce3d640bb38d3e51888e to: /var/lib/boinc/slots/4/oifs_43r3_ps_2530_2021050100_123_990_12206764.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/4/oifs_43r3_ps_2530_2021050100_123_990_12206764.zip
ic_ancil_file: ic_ancil_12206764
ifsdata_file: ifsdata_12206764
climate_data_file: clim_data_12206764
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_43854f33bbd5e4731ac504454f342a2c to: /var/lib/boinc/slots/4/ic_ancil_12206764.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/4/ic_ancil_12206764.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_6c632e10e52988d7cc3ff9c2ef4a6af9 to: /var/lib/boinc/slots/4/ifsdata/ifsdata_12206764.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/4/ifsdata/ifsdata_12206764.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/4/159l_2/clim_data_12206764.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/4/159l_2/clim_data_12206764.zip
Checking for progress XML file: /var/lib/boinc/slots/4/progress_file_12206764.xml
Opened progress file ok : /var/lib/boinc/slots/4/progress_file_12206764.xml
-- Model is restarting --
Adjusting last_iter, 1875, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/4/progress_file_12206764.xml
last_cpu_time: 39471
upload_file_number: 78
last_iter: 1873
last_upload: 6739200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964
The child process has been launched with process id: 4484
Executing the command: /var/lib/boinc/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fa0
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x815590 (old at 0x1ce8cf0)
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x815590 (old at (nil))
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x815590 (old at 0x1ce8cf0)
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x815590 (old at (nil))
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x815590 (old at 0x1ce8cf0)
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x815590 (old at 0x1ce8cf0)
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x815590 (old at (nil))
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x815590 (old at 0x1ce8cf0)
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x815590 (old at 0x1ce8cf0)
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x815590 (old at 0x1ce8cf0)
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x815590 (old at (nil))
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x815590 (old at (nil))
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x815590 (old at (nil))
[EC_DRHOOK:science:1:1:4484:4484] [20230209:203440:1675996480:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:35:26 STEP 1872 H=1872:00 +CPU= 29.991
  20:35:43 STEP 1873 H=1873:00 +CPU= 17.214
  20:36:01 STEP 1874 H=1874:00 +CPU= 17.750
  20:36:28 STEP 1875 H=1875:00 +CPU= 26.983
  20:36:46 STEP 1876 H=1876:00 +CPU= 17.803
  20:37:03 STEP 1877 H=1877:00 +CPU= 17.206
  20:37:31 STEP 1878 H=1878:00 +CPU= 27.578
  20:37:48 STEP 1879 H=1879:00 +CPU= 17.184
  20:38:06 STEP 1880 H=1880:00 +CPU= 17.707
  20:38:33 STEP 1881 H=1881:00 +CPU= 26.996
  20:38:51 STEP 1882 H=1882:00 +CPU= 17.702
  20:39:08 STEP 1883 H=1883:00 +CPU= 17.202
  20:39:38 STEP 1884 H=1884:00 +CPU= 29.890
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001884
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001884
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001884
  20:39:55 STEP 1885 H=1885:00 +CPU= 17.203
  20:40:13 STEP 1886 H=1886:00 +CPU= 17.735
  20:40:40 STEP 1887 H=1887:00 +CPU= 26.939
  20:40:58 STEP 1888 H=1888:00 +CPU= 17.719
  20:41:15 STEP 1889 H=1889:00 +CPU= 17.193
  20:41:43 STEP 1890 H=1890:00 +CPU= 27.514
  20:42:00 STEP 1891 H=1891:00 +CPU= 17.162
  20:42:18 STEP 1892 H=1892:00 +CPU= 17.714
  20:42:45 STEP 1893 H=1893:00 +CPU= 26.874
  20:43:03 STEP 1894 H=1894:00 +CPU= 17.645
  20:43:21 STEP 1895 H=1895:00 +CPU= 17.706
  20:43:52 STEP 1896 H=1896:00 +CPU= 31.581
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001896
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001896
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001884
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_78.zip
Uploading the intermediate file: upload_file_78.zip
  20:44:10 STEP 1897 H=1897:00 +CPU= 17.096
  20:44:28 STEP 1898 H=1898:00 +CPU= 17.679
  20:44:55 STEP 1899 H=1899:00 +CPU= 26.923
  20:45:13 STEP 1900 H=1900:00 +CPU= 17.687
  20:45:30 STEP 1901 H=1901:00 +CPU= 17.180
  20:45:58 STEP 1902 H=1902:00 +CPU= 27.462
  20:46:15 STEP 1903 H=1903:00 +CPU= 17.195
  20:46:33 STEP 1904 H=1904:00 +CPU= 17.783
  20:47:00 STEP 1905 H=1905:00 +CPU= 26.876
  20:47:18 STEP 1906 H=1906:00 +CPU= 17.728
  20:47:35 STEP 1907 H=1907:00 +CPU= 17.205
  20:48:05 STEP 1908 H=1908:00 +CPU= 29.864
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001908
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001908
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001908
  20:48:22 STEP 1909 H=1909:00 +CPU= 17.201
  20:48:40 STEP 1910 H=1910:00 +CPU= 17.706
  20:49:07 STEP 1911 H=1911:00 +CPU= 26.853
  20:49:25 STEP 1912 H=1912:00 +CPU= 17.684
  20:49:42 STEP 1913 H=1913:00 +CPU= 17.172
  20:50:10 STEP 1914 H=1914:00 +CPU= 27.415
  20:50:27 STEP 1915 H=1915:00 +CPU= 17.135
  20:50:45 STEP 1916 H=1916:00 +CPU= 17.668
  20:51:12 STEP 1917 H=1917:00 +CPU= 26.842
  20:51:29 STEP 1918 H=1918:00 +CPU= 17.641
  20:51:47 STEP 1919 H=1919:00 +CPU= 17.691
  20:52:19 STEP 1920 H=1920:00 +CPU= 31.574
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001920
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001920
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001908
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_79.zip
Uploading the intermediate file: upload_file_79.zip
  20:52:36 STEP 1921 H=1921:00 +CPU= 17.065
  20:52:55 STEP 1922 H=1922:00 +CPU= 17.626
  20:53:22 STEP 1923 H=1923:00 +CPU= 26.829
  20:53:39 STEP 1924 H=1924:00 +CPU= 17.642
  20:53:56 STEP 1925 H=1925:00 +CPU= 17.113
  20:54:24 STEP 1926 H=1926:00 +CPU= 27.444
  20:54:41 STEP 1927 H=1927:00 +CPU= 17.155
  20:54:59 STEP 1928 H=1928:00 +CPU= 17.677
  20:55:26 STEP 1929 H=1929:00 +CPU= 26.865
  20:55:44 STEP 1930 H=1930:00 +CPU= 17.685
  20:56:01 STEP 1931 H=1931:00 +CPU= 17.154
  20:56:31 STEP 1932 H=1932:00 +CPU= 29.886
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001932
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001932
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001932
  20:56:48 STEP 1933 H=1933:00 +CPU= 17.167
  20:57:06 STEP 1934 H=1934:00 +CPU= 17.682
  20:57:33 STEP 1935 H=1935:00 +CPU= 26.857
  20:57:51 STEP 1936 H=1936:00 +CPU= 17.716
  20:58:08 STEP 1937 H=1937:00 +CPU= 17.132
  20:58:36 STEP 1938 H=1938:00 +CPU= 27.485
  20:58:53 STEP 1939 H=1939:00 +CPU= 17.136
  20:59:11 STEP 1940 H=1940:00 +CPU= 17.664
  20:59:38 STEP 1941 H=1941:00 +CPU= 26.864
  20:59:55 STEP 1942 H=1942:00 +CPU= 17.677
  21:00:13 STEP 1943 H=1943:00 +CPU= 17.733
  21:00:45 STEP 1944 H=1944:00 +CPU= 31.607
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001944
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001944
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001932
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_80.zip
Uploading the intermediate file: upload_file_80.zip
  21:01:03 STEP 1945 H=1945:00 +CPU= 17.086
  21:01:21 STEP 1946 H=1946:00 +CPU= 17.682
  21:01:48 STEP 1947 H=1947:00 +CPU= 26.937
  21:02:06 STEP 1948 H=1948:00 +CPU= 17.698
  21:02:23 STEP 1949 H=1949:00 +CPU= 17.157
  21:02:50 STEP 1950 H=1950:00 +CPU= 27.470
  21:03:08 STEP 1951 H=1951:00 +CPU= 17.184
  21:03:26 STEP 1952 H=1952:00 +CPU= 17.712
  21:03:53 STEP 1953 H=1953:00 +CPU= 26.960
  21:04:10 STEP 1954 H=1954:00 +CPU= 17.745
  21:04:28 STEP 1955 H=1955:00 +CPU= 17.187
  21:04:58 STEP 1956 H=1956:00 +CPU= 29.872
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001956
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001956
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001956
  21:05:15 STEP 1957 H=1957:00 +CPU= 17.173
  21:05:33 STEP 1958 H=1958:00 +CPU= 17.701
  21:06:00 STEP 1959 H=1959:00 +CPU= 26.851
  21:06:18 STEP 1960 H=1960:00 +CPU= 17.704
  21:06:35 STEP 1961 H=1961:00 +CPU= 17.175
  21:07:02 STEP 1962 H=1962:00 +CPU= 27.457
  21:07:20 STEP 1963 H=1963:00 +CPU= 17.156
  21:07:37 STEP 1964 H=1964:00 +CPU= 17.700
  21:08:04 STEP 1965 H=1965:00 +CPU= 26.871
  21:08:22 STEP 1966 H=1966:00 +CPU= 17.694
  21:08:40 STEP 1967 H=1967:00 +CPU= 17.730
  21:09:12 STEP 1968 H=1968:00 +CPU= 31.600
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001968
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001968
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001956
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_81.zip
Uploading the intermediate file: upload_file_81.zip
  21:09:30 STEP 1969 H=1969:00 +CPU= 17.115
  21:09:48 STEP 1970 H=1970:00 +CPU= 17.691
  21:10:15 STEP 1971 H=1971:00 +CPU= 26.897
  21:10:33 STEP 1972 H=1972:00 +CPU= 17.707
  21:10:50 STEP 1973 H=1973:00 +CPU= 17.204
  21:11:17 STEP 1974 H=1974:00 +CPU= 27.505
  21:11:35 STEP 1975 H=1975:00 +CPU= 17.204
  21:11:53 STEP 1976 H=1976:00 +CPU= 17.747
  21:12:20 STEP 1977 H=1977:00 +CPU= 26.899
  21:12:37 STEP 1978 H=1978:00 +CPU= 17.729
  21:12:55 STEP 1979 H=1979:00 +CPU= 17.186
  21:13:25 STEP 1980 H=1980:00 +CPU= 29.944
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001980
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001980
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001980
  21:13:42 STEP 1981 H=1981:00 +CPU= 17.232
  21:14:00 STEP 1982 H=1982:00 +CPU= 17.762
  21:14:27 STEP 1983 H=1983:00 +CPU= 26.947
  21:14:45 STEP 1984 H=1984:00 +CPU= 17.736
  21:15:02 STEP 1985 H=1985:00 +CPU= 17.192
  21:15:30 STEP 1986 H=1986:00 +CPU= 27.476
  21:15:47 STEP 1987 H=1987:00 +CPU= 17.201
  21:16:05 STEP 1988 H=1988:00 +CPU= 17.702
  21:16:32 STEP 1989 H=1989:00 +CPU= 26.861
  21:16:49 STEP 1990 H=1990:00 +CPU= 17.683
  21:17:07 STEP 1991 H=1991:00 +CPU= 17.762
  21:17:39 STEP 1992 H=1992:00 +CPU= 31.629
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+001992
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+001992
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001980
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_82.zip
Uploading the intermediate file: upload_file_82.zip
  21:17:57 STEP 1993 H=1993:00 +CPU= 17.131
  21:18:15 STEP 1994 H=1994:00 +CPU= 17.704
  21:18:42 STEP 1995 H=1995:00 +CPU= 26.885
  21:19:00 STEP 1996 H=1996:00 +CPU= 17.714
  21:19:17 STEP 1997 H=1997:00 +CPU= 17.170
  21:19:45 STEP 1998 H=1998:00 +CPU= 27.528
  21:20:02 STEP 1999 H=1999:00 +CPU= 17.186
  21:20:20 STEP 2000 H=2000:00 +CPU= 17.742
  21:20:47 STEP 2001 H=2001:00 +CPU= 26.943
  21:21:05 STEP 2002 H=2002:00 +CPU= 17.715
  21:21:22 STEP 2003 H=2003:00 +CPU= 17.185
  21:21:52 STEP 2004 H=2004:00 +CPU= 29.963
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002004
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002004
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002004
  21:22:09 STEP 2005 H=2005:00 +CPU= 17.176
  21:22:27 STEP 2006 H=2006:00 +CPU= 17.715
  21:22:54 STEP 2007 H=2007:00 +CPU= 26.901
  21:23:12 STEP 2008 H=2008:00 +CPU= 17.706
  21:23:29 STEP 2009 H=2009:00 +CPU= 17.185
  21:23:57 STEP 2010 H=2010:00 +CPU= 27.544
  21:24:14 STEP 2011 H=2011:00 +CPU= 17.172
  21:24:32 STEP 2012 H=2012:00 +CPU= 17.698
  21:24:59 STEP 2013 H=2013:00 +CPU= 26.909
  21:25:17 STEP 2014 H=2014:00 +CPU= 17.707
  21:25:35 STEP 2015 H=2015:00 +CPU= 17.766
  21:26:07 STEP 2016 H=2016:00 +CPU= 31.695
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002016
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002016
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002004
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_83.zip
Uploading the intermediate file: upload_file_83.zip
  21:26:25 STEP 2017 H=2017:00 +CPU= 17.120
  21:26:43 STEP 2018 H=2018:00 +CPU= 17.750
  21:27:10 STEP 2019 H=2019:00 +CPU= 26.985
  21:27:27 STEP 2020 H=2020:00 +CPU= 17.729
  21:27:45 STEP 2021 H=2021:00 +CPU= 17.211
  21:28:12 STEP 2022 H=2022:00 +CPU= 27.586
  21:28:30 STEP 2023 H=2023:00 +CPU= 17.275
  21:28:48 STEP 2024 H=2024:00 +CPU= 17.827
  21:29:15 STEP 2025 H=2025:00 +CPU= 27.032
  21:29:33 STEP 2026 H=2026:00 +CPU= 17.797
  21:29:50 STEP 2027 H=2027:00 +CPU= 17.291
  21:30:20 STEP 2028 H=2028:00 +CPU= 30.072
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002028
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002028
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002028
  21:30:38 STEP 2029 H=2029:00 +CPU= 17.322
  21:30:56 STEP 2030 H=2030:00 +CPU= 17.871
  21:31:23 STEP 2031 H=2031:00 +CPU= 27.063
  21:31:41 STEP 2032 H=2032:00 +CPU= 17.802
  21:31:58 STEP 2033 H=2033:00 +CPU= 17.297
  21:32:26 STEP 2034 H=2034:00 +CPU= 27.638
  21:32:43 STEP 2035 H=2035:00 +CPU= 17.398
  21:33:01 STEP 2036 H=2036:00 +CPU= 17.920
  21:33:29 STEP 2037 H=2037:00 +CPU= 27.084
  21:33:47 STEP 2038 H=2038:00 +CPU= 17.821
  21:34:05 STEP 2039 H=2039:00 +CPU= 17.959
  21:34:37 STEP 2040 H=2040:00 +CPU= 31.931
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002040
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002040
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002028
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_84.zip
Uploading the intermediate file: upload_file_84.zip
  21:34:55 STEP 2041 H=2041:00 +CPU= 17.282
  21:35:13 STEP 2042 H=2042:00 +CPU= 17.831
  21:35:40 STEP 2043 H=2043:00 +CPU= 27.042
  21:35:58 STEP 2044 H=2044:00 +CPU= 17.787
  21:36:15 STEP 2045 H=2045:00 +CPU= 17.278
  21:36:43 STEP 2046 H=2046:00 +CPU= 27.702
  21:37:01 STEP 2047 H=2047:00 +CPU= 17.288
  21:37:18 STEP 2048 H=2048:00 +CPU= 17.818
  21:37:46 STEP 2049 H=2049:00 +CPU= 27.088
  21:38:03 STEP 2050 H=2050:00 +CPU= 17.805
  21:38:21 STEP 2051 H=2051:00 +CPU= 17.289
  21:38:51 STEP 2052 H=2052:00 +CPU= 30.137
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002052
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002052
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002052
  21:39:09 STEP 2053 H=2053:00 +CPU= 17.271
  21:39:27 STEP 2054 H=2054:00 +CPU= 17.814
  21:39:54 STEP 2055 H=2055:00 +CPU= 27.044
  21:40:12 STEP 2056 H=2056:00 +CPU= 17.785
  21:40:29 STEP 2057 H=2057:00 +CPU= 17.276
  21:40:57 STEP 2058 H=2058:00 +CPU= 27.637
  21:41:14 STEP 2059 H=2059:00 +CPU= 17.244
  21:41:32 STEP 2060 H=2060:00 +CPU= 17.758
  21:42:00 STEP 2061 H=2061:00 +CPU= 26.987
  21:42:18 STEP 2062 H=2062:00 +CPU= 17.737
  21:42:35 STEP 2063 H=2063:00 +CPU= 17.690
  21:43:07 STEP 2064 H=2064:00 +CPU= 31.727
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002064
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002064
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002052
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_85.zip
Uploading the intermediate file: upload_file_85.zip
  21:43:25 STEP 2065 H=2065:00 +CPU= 17.148
  21:43:43 STEP 2066 H=2066:00 +CPU= 17.745
  21:44:10 STEP 2067 H=2067:00 +CPU= 26.963
  21:44:28 STEP 2068 H=2068:00 +CPU= 17.742
  21:44:46 STEP 2069 H=2069:00 +CPU= 17.226
  21:45:13 STEP 2070 H=2070:00 +CPU= 27.623
  21:45:31 STEP 2071 H=2071:00 +CPU= 17.234
  21:45:48 STEP 2072 H=2072:00 +CPU= 17.771
  21:46:16 STEP 2073 H=2073:00 +CPU= 27.032
  21:46:33 STEP 2074 H=2074:00 +CPU= 17.752
  21:46:51 STEP 2075 H=2075:00 +CPU= 17.188
  21:47:21 STEP 2076 H=2076:00 +CPU= 30.155
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002076
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002076
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002076
  21:47:38 STEP 2077 H=2077:00 +CPU= 17.248
  21:47:56 STEP 2078 H=2078:00 +CPU= 17.788
  21:48:23 STEP 2079 H=2079:00 +CPU= 26.981
  21:48:41 STEP 2080 H=2080:00 +CPU= 17.771
  21:48:59 STEP 2081 H=2081:00 +CPU= 17.287
  21:49:26 STEP 2082 H=2082:00 +CPU= 27.610
  21:49:44 STEP 2083 H=2083:00 +CPU= 17.257
  21:50:02 STEP 2084 H=2084:00 +CPU= 17.776
  21:50:29 STEP 2085 H=2085:00 +CPU= 26.980
  21:50:46 STEP 2086 H=2086:00 +CPU= 17.747
  21:51:04 STEP 2087 H=2087:00 +CPU= 17.747
  21:51:37 STEP 2088 H=2088:00 +CPU= 31.813
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002088
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002088
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002076
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_86.zip
Uploading the intermediate file: upload_file_86.zip
  21:51:55 STEP 2089 H=2089:00 +CPU= 17.174
  21:52:13 STEP 2090 H=2090:00 +CPU= 17.773
  21:52:40 STEP 2091 H=2091:00 +CPU= 27.003
  21:52:57 STEP 2092 H=2092:00 +CPU= 17.766
  21:53:15 STEP 2093 H=2093:00 +CPU= 17.251
  21:53:43 STEP 2094 H=2094:00 +CPU= 27.588
  21:54:00 STEP 2095 H=2095:00 +CPU= 17.252
  21:54:18 STEP 2096 H=2096:00 +CPU= 17.821
  21:54:45 STEP 2097 H=2097:00 +CPU= 27.005
  21:55:03 STEP 2098 H=2098:00 +CPU= 17.794
  21:55:20 STEP 2099 H=2099:00 +CPU= 17.291
  21:55:50 STEP 2100 H=2100:00 +CPU= 29.980
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002100
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002100
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002100
  21:56:08 STEP 2101 H=2101:00 +CPU= 17.284
  21:56:25 STEP 2102 H=2102:00 +CPU= 17.775
  21:56:53 STEP 2103 H=2103:00 +CPU= 27.029
  21:57:10 STEP 2104 H=2104:00 +CPU= 17.779
  21:57:28 STEP 2105 H=2105:00 +CPU= 17.290
  21:57:56 STEP 2106 H=2106:00 +CPU= 27.599
  21:58:13 STEP 2107 H=2107:00 +CPU= 17.258
  21:58:31 STEP 2108 H=2108:00 +CPU= 17.735
  21:58:58 STEP 2109 H=2109:00 +CPU= 27.057
  21:59:16 STEP 2110 H=2110:00 +CPU= 17.773
  21:59:34 STEP 2111 H=2111:00 +CPU= 17.871
  22:00:06 STEP 2112 H=2112:00 +CPU= 31.735
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002112
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002112
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002100
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_87.zip
Uploading the intermediate file: upload_file_87.zip
  22:00:24 STEP 2113 H=2113:00 +CPU= 17.232
Uploading trickle at timestep: 7603200
  22:00:42 STEP 2114 H=2114:00 +CPU= 17.753
  22:01:09 STEP 2115 H=2115:00 +CPU= 27.044
  22:01:27 STEP 2116 H=2116:00 +CPU= 17.805
  22:01:44 STEP 2117 H=2117:00 +CPU= 17.240
  22:02:12 STEP 2118 H=2118:00 +CPU= 27.638
  22:02:29 STEP 2119 H=2119:00 +CPU= 17.321
  22:02:47 STEP 2120 H=2120:00 +CPU= 17.771
  22:03:14 STEP 2121 H=2121:00 +CPU= 27.078
  22:03:32 STEP 2122 H=2122:00 +CPU= 17.826
  22:03:49 STEP 2123 H=2123:00 +CPU= 17.328
  22:04:20 STEP 2124 H=2124:00 +CPU= 30.012
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002124
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002124
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002124
  22:04:37 STEP 2125 H=2125:00 +CPU= 17.322
  22:04:55 STEP 2126 H=2126:00 +CPU= 17.771
  22:05:22 STEP 2127 H=2127:00 +CPU= 27.055
  22:05:40 STEP 2128 H=2128:00 +CPU= 17.843
  22:05:57 STEP 2129 H=2129:00 +CPU= 17.267
  22:06:25 STEP 2130 H=2130:00 +CPU= 27.634
  22:06:42 STEP 2131 H=2131:00 +CPU= 17.307
  22:07:00 STEP 2132 H=2132:00 +CPU= 17.775
  22:07:27 STEP 2133 H=2133:00 +CPU= 27.017
  22:07:45 STEP 2134 H=2134:00 +CPU= 17.787
  22:08:03 STEP 2135 H=2135:00 +CPU= 17.866
  22:08:35 STEP 2136 H=2136:00 +CPU= 31.644
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002136
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002136
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002124
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_88.zip
Uploading the intermediate file: upload_file_88.zip
  22:08:53 STEP 2137 H=2137:00 +CPU= 17.223
  22:09:11 STEP 2138 H=2138:00 +CPU= 17.732
  22:09:38 STEP 2139 H=2139:00 +CPU= 27.042
  22:09:56 STEP 2140 H=2140:00 +CPU= 17.815
  22:10:13 STEP 2141 H=2141:00 +CPU= 17.244
  22:10:41 STEP 2142 H=2142:00 +CPU= 27.669
  22:10:58 STEP 2143 H=2143:00 +CPU= 17.291
  22:11:16 STEP 2144 H=2144:00 +CPU= 17.767
  22:11:43 STEP 2145 H=2145:00 +CPU= 27.079
  22:12:01 STEP 2146 H=2146:00 +CPU= 17.826
  22:12:19 STEP 2147 H=2147:00 +CPU= 17.261
  22:12:49 STEP 2148 H=2148:00 +CPU= 30.024
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002148
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002148
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002148
  22:13:06 STEP 2149 H=2149:00 +CPU= 17.304
  22:13:24 STEP 2150 H=2150:00 +CPU= 17.773
  22:13:51 STEP 2151 H=2151:00 +CPU= 27.055
  22:14:09 STEP 2152 H=2152:00 +CPU= 17.764
  22:14:26 STEP 2153 H=2153:00 +CPU= 17.203
  22:14:54 STEP 2154 H=2154:00 +CPU= 27.644
  22:15:12 STEP 2155 H=2155:00 +CPU= 17.311
  22:15:29 STEP 2156 H=2156:00 +CPU= 17.723
  22:15:57 STEP 2157 H=2157:00 +CPU= 27.135
  22:16:14 STEP 2158 H=2158:00 +CPU= 17.762
  22:16:33 STEP 2159 H=2159:00 +CPU= 17.870
  22:17:05 STEP 2160 H=2160:00 +CPU= 31.783
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGhpi1+002160
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHhpi1+002160
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMGGhpi1+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMSHhpi1+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12206764/ICMUAhpi1+002148
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_2530_2021050100_123_990_12206764_0_r108912964_89.zip
Uploading the intermediate file: upload_file_89.zip
  22:17:23 STEP 2161 H=2161:00 +CPU= 17.194
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:50:23 1536819 22306913 oifs_43r3_ps_2530_2021050100_123_990_12206764_0 7,603,200 44,553 0.0059
09 May 2023 16:50:21 1536819 22306913 oifs_43r3_ps_2530_2021050100_123_990_12206764_0 6,048,000 35,393 0.0059
09 May 2023 16:50:19 1536819 22306913 oifs_43r3_ps_2530_2021050100_123_990_12206764_0 5,184,000 30,354 0.0059
09 May 2023 16:50:18 1536819 22306913 oifs_43r3_ps_2530_2021050100_123_990_12206764_0 4,320,000 25,315 0.0059
09 May 2023 16:50:16 1536819 22306913 oifs_43r3_ps_2530_2021050100_123_990_12206764_0 3,456,000 20,277 0.0059
09 May 2023 16:50:15 1536819 22306913 oifs_43r3_ps_2530_2021050100_123_990_12206764_0 2,592,000 15,229 0.0059
09 May 2023 16:50:14 1536819 22306913 oifs_43r3_ps_2530_2021050100_123_990_12206764_0 1,728,000 10,190 0.0059
09 May 2023 16:50:12 1536819 22306913 oifs_43r3_ps_2530_2021050100_123_990_12206764_0 864,000 5,142 0.0060


©2024 climateprediction.net