climateprediction.net home page
Task 22304416

Task 22304416

Name oifs_43r3_ps_0486_2019050100_123_988_12205130_2
Workunit 12205130
Created 28 Jan 2023, 20:49:16 UTC
Sent 28 Jan 2023, 20:50:28 UTC
Report deadline 29 Mar 2023, 20:50:28 UTC
Received 8 Feb 2023, 21:27:48 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1537759
Run time 20 hours 47 min 37 sec
CPU time 19 hours 40 min 11 sec
Validate state Invalid
Credit 10,886.40
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.63 MB
Peak swap size 4,974.80 MB
Peak disk usage 1,405.28 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
  12:41:55 STEP 1615 H=1615:00 +CPU= 25.716
  12:42:23 STEP 1616 H=1616:00 +CPU= 27.250
  12:43:05 STEP 1617 H=1617:00 +CPU= 41.389
  12:43:33 STEP 1618 H=1618:00 +CPU= 27.039
  12:44:00 STEP 1619 H=1619:00 +CPU= 25.880
  12:44:47 STEP 1620 H=1620:00 +CPU= 45.947
Moving to projects directory: /config/slots/8/ICMGGhq0f+001620
Moving to projects directory: /config/slots/8/ICMSHhq0f+001620
Moving to projects directory: /config/slots/8/ICMUAhq0f+001620
  12:45:14 STEP 1621 H=1621:00 +CPU= 25.818
  12:45:42 STEP 1622 H=1622:00 +CPU= 27.025
  12:46:25 STEP 1623 H=1623:00 +CPU= 41.779
  12:46:53 STEP 1624 H=1624:00 +CPU= 26.707
  12:47:21 STEP 1625 H=1625:00 +CPU= 26.371
  12:48:04 STEP 1626 H=1626:00 +CPU= 42.373
  12:48:31 STEP 1627 H=1627:00 +CPU= 25.915
  12:48:59 STEP 1628 H=1628:00 +CPU= 26.775
  12:49:42 STEP 1629 H=1629:00 +CPU= 41.210
  12:50:11 STEP 1630 H=1630:00 +CPU= 26.986
  12:50:39 STEP 1631 H=1631:00 +CPU= 26.740
  12:51:30 STEP 1632 H=1632:00 +CPU= 48.757
Moving to projects directory: /config/slots/8/ICMGGhq0f+001632
Moving to projects directory: /config/slots/8/ICMSHhq0f+001632
Moving to projects directory: /config/slots/8/ICMUAhq0f+001632
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001608
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001608
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001608
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001620
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001620
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001620
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_67.zip
Uploading the intermediate file: upload_file_67.zip
  12:51:56 STEP 1633 H=1633:00 +CPU= 25.816
  12:52:24 STEP 1634 H=1634:00 +CPU= 27.190
  12:53:07 STEP 1635 H=1635:00 +CPU= 41.353
  12:53:35 STEP 1636 H=1636:00 +CPU= 27.013
  12:54:02 STEP 1637 H=1637:00 +CPU= 25.806
  12:54:46 STEP 1638 H=1638:00 +CPU= 42.515
  12:55:13 STEP 1639 H=1639:00 +CPU= 25.962
  12:55:41 STEP 1640 H=1640:00 +CPU= 27.028
  12:56:23 STEP 1641 H=1641:00 +CPU= 41.347
  12:56:52 STEP 1642 H=1642:00 +CPU= 27.137
  12:57:19 STEP 1643 H=1643:00 +CPU= 26.052
  12:58:06 STEP 1644 H=1644:00 +CPU= 45.834
Moving to projects directory: /config/slots/8/ICMGGhq0f+001644
Moving to projects directory: /config/slots/8/ICMSHhq0f+001644
Moving to projects directory: /config/slots/8/ICMUAhq0f+001644
  12:58:34 STEP 1645 H=1645:00 +CPU= 26.072
  12:59:02 STEP 1646 H=1646:00 +CPU= 27.111
  12:59:44 STEP 1647 H=1647:00 +CPU= 41.144
  13:00:12 STEP 1648 H=1648:00 +CPU= 26.654
  13:00:38 STEP 1649 H=1649:00 +CPU= 25.840
  13:01:21 STEP 1650 H=1650:00 +CPU= 42.000
  13:01:48 STEP 1651 H=1651:00 +CPU= 26.109
  13:02:16 STEP 1652 H=1652:00 +CPU= 26.721
  13:02:59 STEP 1653 H=1653:00 +CPU= 41.357
  13:03:27 STEP 1654 H=1654:00 +CPU= 26.530
  13:03:55 STEP 1655 H=1655:00 +CPU= 26.825
  13:04:45 STEP 1656 H=1656:00 +CPU= 48.742
Moving to projects directory: /config/slots/8/ICMGGhq0f+001656
Moving to projects directory: /config/slots/8/ICMSHhq0f+001656
Moving to projects directory: /config/slots/8/ICMUAhq0f+001656
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001632
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001632
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001632
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001644
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001644
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001644
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_68.zip
Uploading the intermediate file: upload_file_68.zip
  13:05:14 STEP 1657 H=1657:00 +CPU= 26.139
  13:05:42 STEP 1658 H=1658:00 +CPU= 26.941
  13:06:24 STEP 1659 H=1659:00 +CPU= 41.396
  13:06:52 STEP 1660 H=1660:00 +CPU= 26.652
  13:07:19 STEP 1661 H=1661:00 +CPU= 26.312
  13:08:03 STEP 1662 H=1662:00 +CPU= 42.425
  13:08:29 STEP 1663 H=1663:00 +CPU= 25.859
  13:08:57 STEP 1664 H=1664:00 +CPU= 26.678
  13:09:40 STEP 1665 H=1665:00 +CPU= 41.263
  13:10:08 STEP 1666 H=1666:00 +CPU= 27.217
  13:10:34 STEP 1667 H=1667:00 +CPU= 25.725
  13:11:22 STEP 1668 H=1668:00 +CPU= 46.194
Moving to projects directory: /config/slots/8/ICMGGhq0f+001668
Moving to projects directory: /config/slots/8/ICMSHhq0f+001668
Moving to projects directory: /config/slots/8/ICMUAhq0f+001668
  13:11:49 STEP 1669 H=1669:00 +CPU= 25.878
  13:12:17 STEP 1670 H=1670:00 +CPU= 26.815
  13:12:59 STEP 1671 H=1671:00 +CPU= 41.239
  13:13:27 STEP 1672 H=1672:00 +CPU= 27.063
  13:13:54 STEP 1673 H=1673:00 +CPU= 25.973
  13:14:37 STEP 1674 H=1674:00 +CPU= 42.302
  13:15:04 STEP 1675 H=1675:00 +CPU= 25.962
  13:15:32 STEP 1676 H=1676:00 +CPU= 26.806
  13:16:15 STEP 1677 H=1677:00 +CPU= 41.670
  13:16:43 STEP 1678 H=1678:00 +CPU= 26.789
  13:17:10 STEP 1679 H=1679:00 +CPU= 26.851
  13:18:01 STEP 1680 H=1680:00 +CPU= 49.238
Moving to projects directory: /config/slots/8/ICMGGhq0f+001680
Moving to projects directory: /config/slots/8/ICMSHhq0f+001680
Moving to projects directory: /config/slots/8/ICMUAhq0f+001680
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001656
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001656
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001656
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001668
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001668
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001668
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_69.zip
Uploading the intermediate file: upload_file_69.zip
Uploading trickle at timestep: 6048000
  13:18:28 STEP 1681 H=1681:00 +CPU= 26.208
  13:18:56 STEP 1682 H=1682:00 +CPU= 26.876
  13:19:39 STEP 1683 H=1683:00 +CPU= 41.387
  13:20:07 STEP 1684 H=1684:00 +CPU= 26.767
  13:20:34 STEP 1685 H=1685:00 +CPU= 25.938
  13:21:18 STEP 1686 H=1686:00 +CPU= 42.489
  13:21:45 STEP 1687 H=1687:00 +CPU= 26.173
  13:22:13 STEP 1688 H=1688:00 +CPU= 27.016
  13:22:56 STEP 1689 H=1689:00 +CPU= 41.071
  13:23:23 STEP 1690 H=1690:00 +CPU= 26.596
  13:23:50 STEP 1691 H=1691:00 +CPU= 25.977
  13:24:38 STEP 1692 H=1692:00 +CPU= 46.187
Moving to projects directory: /config/slots/8/ICMGGhq0f+001692
Moving to projects directory: /config/slots/8/ICMSHhq0f+001692
Moving to projects directory: /config/slots/8/ICMUAhq0f+001692
  13:25:04 STEP 1693 H=1693:00 +CPU= 25.930
  13:25:32 STEP 1694 H=1694:00 +CPU= 26.788
  13:26:15 STEP 1695 H=1695:00 +CPU= 41.205
  13:26:42 STEP 1696 H=1696:00 +CPU= 26.772
  13:27:10 STEP 1697 H=1697:00 +CPU= 26.613
  13:27:54 STEP 1698 H=1698:00 +CPU= 42.474
  13:28:22 STEP 1699 H=1699:00 +CPU= 26.220
  13:28:49 STEP 1700 H=1700:00 +CPU= 26.723
  13:29:32 STEP 1701 H=1701:00 +CPU= 41.251
  13:30:00 STEP 1702 H=1702:00 +CPU= 27.032
  13:30:28 STEP 1703 H=1703:00 +CPU= 27.086
  13:31:19 STEP 1704 H=1704:00 +CPU= 49.127
Moving to projects directory: /config/slots/8/ICMGGhq0f+001704
Moving to projects directory: /config/slots/8/ICMSHhq0f+001704
Moving to projects directory: /config/slots/8/ICMUAhq0f+001704
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001680
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001680
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001680
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001692
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001692
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001692
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_70.zip
Uploading the intermediate file: upload_file_70.zip
  13:31:46 STEP 1705 H=1705:00 +CPU= 26.241
  13:32:14 STEP 1706 H=1706:00 +CPU= 26.791
  13:32:57 STEP 1707 H=1707:00 +CPU= 41.300
  13:33:25 STEP 1708 H=1708:00 +CPU= 27.139
  13:33:52 STEP 1709 H=1709:00 +CPU= 26.071
  13:34:35 STEP 1710 H=1710:00 +CPU= 42.345
  13:35:02 STEP 1711 H=1711:00 +CPU= 26.125
  13:35:30 STEP 1712 H=1712:00 +CPU= 26.940
  13:36:14 STEP 1713 H=1713:00 +CPU= 42.007
  13:36:42 STEP 1714 H=1714:00 +CPU= 27.066
  13:37:09 STEP 1715 H=1715:00 +CPU= 26.084
  13:37:57 STEP 1716 H=1716:00 +CPU= 45.624
Moving to projects directory: /config/slots/8/ICMGGhq0f+001716
Moving to projects directory: /config/slots/8/ICMSHhq0f+001716
Moving to projects directory: /config/slots/8/ICMUAhq0f+001716
  13:38:23 STEP 1717 H=1717:00 +CPU= 25.873
  13:38:51 STEP 1718 H=1718:00 +CPU= 27.015
  13:39:35 STEP 1719 H=1719:00 +CPU= 41.971
  13:40:03 STEP 1720 H=1720:00 +CPU= 26.800
  13:40:29 STEP 1721 H=1721:00 +CPU= 25.754
  13:41:14 STEP 1722 H=1722:00 +CPU= 42.661
  13:41:41 STEP 1723 H=1723:00 +CPU= 26.064
  13:42:08 STEP 1724 H=1724:00 +CPU= 26.711
  13:42:52 STEP 1725 H=1725:00 +CPU= 41.768
  13:43:20 STEP 1726 H=1726:00 +CPU= 26.959
  13:43:48 STEP 1727 H=1727:00 +CPU= 27.210
  13:44:39 STEP 1728 H=1728:00 +CPU= 48.977
Moving to projects directory: /config/slots/8/ICMGGhq0f+001728
Moving to projects directory: /config/slots/8/ICMSHhq0f+001728
Moving to projects directory: /config/slots/8/ICMUAhq0f+001728
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001704
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001704
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001704
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001716
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001716
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001716
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_71.zip
Uploading the intermediate file: upload_file_71.zip
  13:45:06 STEP 1729 H=1729:00 +CPU= 26.017
  13:45:34 STEP 1730 H=1730:00 +CPU= 27.065
  13:46:17 STEP 1731 H=1731:00 +CPU= 41.400
  13:46:46 STEP 1732 H=1732:00 +CPU= 27.169
  13:47:13 STEP 1733 H=1733:00 +CPU= 26.346
  13:47:58 STEP 1734 H=1734:00 +CPU= 43.074
  13:48:25 STEP 1735 H=1735:00 +CPU= 26.269
  13:48:54 STEP 1736 H=1736:00 +CPU= 27.520
  13:49:38 STEP 1737 H=1737:00 +CPU= 42.159
  13:50:07 STEP 1738 H=1738:00 +CPU= 27.750
  13:50:36 STEP 1739 H=1739:00 +CPU= 26.559
  13:51:25 STEP 1740 H=1740:00 +CPU= 47.125
Moving to projects directory: /config/slots/8/ICMGGhq0f+001740
Moving to projects directory: /config/slots/8/ICMSHhq0f+001740
Moving to projects directory: /config/slots/8/ICMUAhq0f+001740
  13:51:54 STEP 1741 H=1741:00 +CPU= 26.681
  13:52:22 STEP 1742 H=1742:00 +CPU= 27.259
  13:53:07 STEP 1743 H=1743:00 +CPU= 42.336
  13:53:34 STEP 1744 H=1744:00 +CPU= 26.974
  13:54:02 STEP 1745 H=1745:00 +CPU= 26.305
  13:54:46 STEP 1746 H=1746:00 +CPU= 43.013
  13:55:14 STEP 1747 H=1747:00 +CPU= 26.514
  13:55:42 STEP 1748 H=1748:00 +CPU= 27.385
  13:56:26 STEP 1749 H=1749:00 +CPU= 41.901
  13:56:54 STEP 1750 H=1750:00 +CPU= 27.570
  13:57:24 STEP 1751 H=1751:00 +CPU= 27.248
  13:58:16 STEP 1752 H=1752:00 +CPU= 49.698
Moving to projects directory: /config/slots/8/ICMGGhq0f+001752
Moving to projects directory: /config/slots/8/ICMSHhq0f+001752
Moving to projects directory: /config/slots/8/ICMUAhq0f+001752
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001728
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001728
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001728
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001740
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001740
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001740
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_72.zip
Uploading the intermediate file: upload_file_72.zip
  13:58:44 STEP 1753 H=1753:00 +CPU= 26.825
  13:59:13 STEP 1754 H=1754:00 +CPU= 27.749
  13:59:57 STEP 1755 H=1755:00 +CPU= 42.098
  14:00:25 STEP 1756 H=1756:00 +CPU= 27.192
  14:00:53 STEP 1757 H=1757:00 +CPU= 26.333
  14:01:37 STEP 1758 H=1758:00 +CPU= 43.039
  14:02:05 STEP 1759 H=1759:00 +CPU= 26.355
  14:02:33 STEP 1760 H=1760:00 +CPU= 27.542
  14:03:16 STEP 1761 H=1761:00 +CPU= 41.670
  14:03:45 STEP 1762 H=1762:00 +CPU= 27.304
  14:04:12 STEP 1763 H=1763:00 +CPU= 26.510
  14:05:01 STEP 1764 H=1764:00 +CPU= 46.834
Moving to projects directory: /config/slots/8/ICMGGhq0f+001764
Moving to projects directory: /config/slots/8/ICMSHhq0f+001764
Moving to projects directory: /config/slots/8/ICMUAhq0f+001764
  14:05:31 STEP 1765 H=1765:00 +CPU= 26.809
  14:06:00 STEP 1766 H=1766:00 +CPU= 27.749
  14:06:44 STEP 1767 H=1767:00 +CPU= 41.728
  14:07:12 STEP 1768 H=1768:00 +CPU= 27.307
  14:07:40 STEP 1769 H=1769:00 +CPU= 26.816
  14:08:25 STEP 1770 H=1770:00 +CPU= 42.929
  14:08:53 STEP 1771 H=1771:00 +CPU= 26.389
  14:09:22 STEP 1772 H=1772:00 +CPU= 27.461
  14:10:06 STEP 1773 H=1773:00 +CPU= 41.914
  14:10:35 STEP 1774 H=1774:00 +CPU= 27.403
  14:11:03 STEP 1775 H=1775:00 +CPU= 27.207
  14:11:55 STEP 1776 H=1776:00 +CPU= 49.252
Moving to projects directory: /config/slots/8/ICMGGhq0f+001776
Moving to projects directory: /config/slots/8/ICMSHhq0f+001776
Moving to projects directory: /config/slots/8/ICMUAhq0f+001776
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001752
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001752
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001752
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001764
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001764
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001764
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_73.zip
Uploading the intermediate file: upload_file_73.zip
  14:12:23 STEP 1777 H=1777:00 +CPU= 26.644
  14:12:51 STEP 1778 H=1778:00 +CPU= 27.396
  14:13:35 STEP 1779 H=1779:00 +CPU= 42.248
  14:14:03 STEP 1780 H=1780:00 +CPU= 27.107
  14:14:31 STEP 1781 H=1781:00 +CPU= 26.604
  14:15:16 STEP 1782 H=1782:00 +CPU= 42.939
  14:15:44 STEP 1783 H=1783:00 +CPU= 26.432
  14:16:13 STEP 1784 H=1784:00 +CPU= 27.535
  14:16:58 STEP 1785 H=1785:00 +CPU= 41.908
  14:17:29 STEP 1786 H=1786:00 +CPU= 27.380
  14:17:57 STEP 1787 H=1787:00 +CPU= 26.829
  14:18:49 STEP 1788 H=1788:00 +CPU= 46.682
Moving to projects directory: /config/slots/8/ICMGGhq0f+001788
Moving to projects directory: /config/slots/8/ICMSHhq0f+001788
Moving to projects directory: /config/slots/8/ICMUAhq0f+001788
  14:19:17 STEP 1789 H=1789:00 +CPU= 26.567
  14:19:47 STEP 1790 H=1790:00 +CPU= 27.147
  14:20:33 STEP 1791 H=1791:00 +CPU= 42.254
  14:21:03 STEP 1792 H=1792:00 +CPU= 27.317
  14:21:33 STEP 1793 H=1793:00 +CPU= 26.522
  14:22:20 STEP 1794 H=1794:00 +CPU= 43.377
  14:22:48 STEP 1795 H=1795:00 +CPU= 26.645
  14:23:18 STEP 1796 H=1796:00 +CPU= 27.423
  14:24:02 STEP 1797 H=1797:00 +CPU= 41.943
  14:24:30 STEP 1798 H=1798:00 +CPU= 27.072
  14:24:58 STEP 1799 H=1799:00 +CPU= 27.295
  14:25:50 STEP 1800 H=1800:00 +CPU= 49.634
Moving to projects directory: /config/slots/8/ICMGGhq0f+001800
Moving to projects directory: /config/slots/8/ICMSHhq0f+001800
Moving to projects directory: /config/slots/8/ICMUAhq0f+001800
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001776
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001776
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001776
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001788
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001788
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001788
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_74.zip
Uploading the intermediate file: upload_file_74.zip
  14:26:17 STEP 1801 H=1801:00 +CPU= 26.293
  14:26:46 STEP 1802 H=1802:00 +CPU= 27.321
  14:27:29 STEP 1803 H=1803:00 +CPU= 42.011
  14:27:58 STEP 1804 H=1804:00 +CPU= 27.375
  14:28:25 STEP 1805 H=1805:00 +CPU= 26.250
  14:29:09 STEP 1806 H=1806:00 +CPU= 42.732
  14:29:37 STEP 1807 H=1807:00 +CPU= 26.437
  14:30:05 STEP 1808 H=1808:00 +CPU= 27.078
  14:30:48 STEP 1809 H=1809:00 +CPU= 42.266
  14:31:17 STEP 1810 H=1810:00 +CPU= 27.348
  14:31:44 STEP 1811 H=1811:00 +CPU= 26.525
  14:32:33 STEP 1812 H=1812:00 +CPU= 46.911
Moving to projects directory: /config/slots/8/ICMGGhq0f+001812
Moving to projects directory: /config/slots/8/ICMSHhq0f+001812
Moving to projects directory: /config/slots/8/ICMUAhq0f+001812
  14:33:00 STEP 1813 H=1813:00 +CPU= 26.390
  14:33:28 STEP 1814 H=1814:00 +CPU= 27.465
  14:34:12 STEP 1815 H=1815:00 +CPU= 41.897
  14:34:40 STEP 1816 H=1816:00 +CPU= 27.198
  14:35:08 STEP 1817 H=1817:00 +CPU= 26.442
  14:35:53 STEP 1818 H=1818:00 +CPU= 43.756
  14:36:21 STEP 1819 H=1819:00 +CPU= 26.547
  14:36:49 STEP 1820 H=1820:00 +CPU= 26.911
  14:37:33 STEP 1821 H=1821:00 +CPU= 42.139
  14:38:01 STEP 1822 H=1822:00 +CPU= 27.316
  14:38:29 STEP 1823 H=1823:00 +CPU= 27.089
  14:39:21 STEP 1824 H=1824:00 +CPU= 49.826
Moving to projects directory: /config/slots/8/ICMGGhq0f+001824
Moving to projects directory: /config/slots/8/ICMSHhq0f+001824
Moving to projects directory: /config/slots/8/ICMUAhq0f+001824
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001800
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001800
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001800
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001812
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001812
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001812
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_75.zip
Uploading the intermediate file: upload_file_75.zip
  14:39:50 STEP 1825 H=1825:00 +CPU= 26.412
  14:40:19 STEP 1826 H=1826:00 +CPU= 27.371
  14:41:03 STEP 1827 H=1827:00 +CPU= 41.848
  14:41:32 STEP 1828 H=1828:00 +CPU= 27.472
  14:41:59 STEP 1829 H=1829:00 +CPU= 26.535
  14:42:45 STEP 1830 H=1830:00 +CPU= 43.656
  14:43:12 STEP 1831 H=1831:00 +CPU= 26.602
  14:43:40 STEP 1832 H=1832:00 +CPU= 27.273
  14:44:24 STEP 1833 H=1833:00 +CPU= 41.947
  14:44:52 STEP 1834 H=1834:00 +CPU= 27.141
  14:45:18 STEP 1835 H=1835:00 +CPU= 25.923
  14:46:06 STEP 1836 H=1836:00 +CPU= 45.943
Moving to projects directory: /config/slots/8/ICMGGhq0f+001836
Moving to projects directory: /config/slots/8/ICMSHhq0f+001836
Moving to projects directory: /config/slots/8/ICMUAhq0f+001836
  14:46:34 STEP 1837 H=1837:00 +CPU= 26.163
  14:47:03 STEP 1838 H=1838:00 +CPU= 27.005
  14:47:46 STEP 1839 H=1839:00 +CPU= 41.916
  14:48:14 STEP 1840 H=1840:00 +CPU= 27.195
  14:48:41 STEP 1841 H=1841:00 +CPU= 26.102
  14:49:26 STEP 1842 H=1842:00 +CPU= 42.943
  14:49:53 STEP 1843 H=1843:00 +CPU= 26.469
  14:50:21 STEP 1844 H=1844:00 +CPU= 26.958
  14:51:05 STEP 1845 H=1845:00 +CPU= 42.207
  14:51:33 STEP 1846 H=1846:00 +CPU= 26.893
  14:52:01 STEP 1847 H=1847:00 +CPU= 27.302
  14:52:53 STEP 1848 H=1848:00 +CPU= 49.237
Moving to projects directory: /config/slots/8/ICMGGhq0f+001848
Moving to projects directory: /config/slots/8/ICMSHhq0f+001848
Moving to projects directory: /config/slots/8/ICMUAhq0f+001848
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001824
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001824
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001824
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001836
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001836
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001836
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_76.zip
Uploading the intermediate file: upload_file_76.zip
  14:53:20 STEP 1849 H=1849:00 +CPU= 26.267
  14:53:48 STEP 1850 H=1850:00 +CPU= 26.941
  14:54:32 STEP 1851 H=1851:00 +CPU= 41.645
  14:55:00 STEP 1852 H=1852:00 +CPU= 26.885
  14:55:27 STEP 1853 H=1853:00 +CPU= 26.154
  14:56:12 STEP 1854 H=1854:00 +CPU= 42.559
  14:56:38 STEP 1855 H=1855:00 +CPU= 25.778
  14:57:06 STEP 1856 H=1856:00 +CPU= 26.840
  14:57:49 STEP 1857 H=1857:00 +CPU= 41.762
  14:58:17 STEP 1858 H=1858:00 +CPU= 27.443
  14:58:44 STEP 1859 H=1859:00 +CPU= 25.905
  14:59:32 STEP 1860 H=1860:00 +CPU= 46.337
Moving to projects directory: /config/slots/8/ICMGGhq0f+001860
Moving to projects directory: /config/slots/8/ICMSHhq0f+001860
Moving to projects directory: /config/slots/8/ICMUAhq0f+001860
  15:00:00 STEP 1861 H=1861:00 +CPU= 26.302
  15:00:28 STEP 1862 H=1862:00 +CPU= 26.952
  15:01:11 STEP 1863 H=1863:00 +CPU= 41.308
  15:01:39 STEP 1864 H=1864:00 +CPU= 27.366
  15:02:06 STEP 1865 H=1865:00 +CPU= 25.975
  15:02:50 STEP 1866 H=1866:00 +CPU= 42.594
  15:03:18 STEP 1867 H=1867:00 +CPU= 26.540
  15:03:46 STEP 1868 H=1868:00 +CPU= 27.122
  15:04:29 STEP 1869 H=1869:00 +CPU= 41.921
  15:04:58 STEP 1870 H=1870:00 +CPU= 27.343
  15:05:28 STEP 1871 H=1871:00 +CPU= 27.069
  15:06:19 STEP 1872 H=1872:00 +CPU= 48.934
Moving to projects directory: /config/slots/8/ICMGGhq0f+001872
Moving to projects directory: /config/slots/8/ICMSHhq0f+001872
Moving to projects directory: /config/slots/8/ICMUAhq0f+001872
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001848
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001848
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001848
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001860
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001860
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001860
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_77.zip
Uploading the intermediate file: upload_file_77.zip
  15:06:46 STEP 1873 H=1873:00 +CPU= 26.504
  15:07:14 STEP 1874 H=1874:00 +CPU= 27.004
  15:07:58 STEP 1875 H=1875:00 +CPU= 42.491
  15:08:26 STEP 1876 H=1876:00 +CPU= 27.048
  15:08:53 STEP 1877 H=1877:00 +CPU= 25.996
  15:09:38 STEP 1878 H=1878:00 +CPU= 42.921
  15:10:05 STEP 1879 H=1879:00 +CPU= 26.005
  15:10:33 STEP 1880 H=1880:00 +CPU= 26.538
  15:11:16 STEP 1881 H=1881:00 +CPU= 41.756
  15:11:45 STEP 1882 H=1882:00 +CPU= 27.212
  15:12:12 STEP 1883 H=1883:00 +CPU= 26.106
  15:12:59 STEP 1884 H=1884:00 +CPU= 46.352
Moving to projects directory: /config/slots/8/ICMGGhq0f+001884
Moving to projects directory: /config/slots/8/ICMSHhq0f+001884
Moving to projects directory: /config/slots/8/ICMUAhq0f+001884
  15:13:27 STEP 1885 H=1885:00 +CPU= 26.256
  15:13:54 STEP 1886 H=1886:00 +CPU= 26.881
  15:14:37 STEP 1887 H=1887:00 +CPU= 41.611
  15:15:05 STEP 1888 H=1888:00 +CPU= 26.799
  15:15:31 STEP 1889 H=1889:00 +CPU= 25.839
  15:16:16 STEP 1890 H=1890:00 +CPU= 42.737
  15:16:43 STEP 1891 H=1891:00 +CPU= 26.195
  15:17:11 STEP 1892 H=1892:00 +CPU= 26.815
  15:17:54 STEP 1893 H=1893:00 +CPU= 41.502
  15:18:23 STEP 1894 H=1894:00 +CPU= 27.438
  15:18:51 STEP 1895 H=1895:00 +CPU= 27.492
  15:19:42 STEP 1896 H=1896:00 +CPU= 49.308
Moving to projects directory: /config/slots/8/ICMGGhq0f+001896
Moving to projects directory: /config/slots/8/ICMSHhq0f+001896
Moving to projects directory: /config/slots/8/ICMUAhq0f+001896
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001872
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001872
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001872
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001884
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001884
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001884
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_78.zip
Uploading the intermediate file: upload_file_78.zip
  15:20:08 STEP 1897 H=1897:00 +CPU= 25.737
  15:20:36 STEP 1898 H=1898:00 +CPU= 27.114
  15:21:20 STEP 1899 H=1899:00 +CPU= 42.025
  15:21:48 STEP 1900 H=1900:00 +CPU= 27.370
  15:22:15 STEP 1901 H=1901:00 +CPU= 26.020
  15:23:00 STEP 1902 H=1902:00 +CPU= 43.110
  15:23:28 STEP 1903 H=1903:00 +CPU= 26.608
  15:23:56 STEP 1904 H=1904:00 +CPU= 26.898
  15:24:39 STEP 1905 H=1905:00 +CPU= 41.638
  15:25:08 STEP 1906 H=1906:00 +CPU= 27.492
  15:25:35 STEP 1907 H=1907:00 +CPU= 26.127
  15:26:23 STEP 1908 H=1908:00 +CPU= 46.256
Moving to projects directory: /config/slots/8/ICMGGhq0f+001908
Moving to projects directory: /config/slots/8/ICMSHhq0f+001908
Moving to projects directory: /config/slots/8/ICMUAhq0f+001908
  15:26:51 STEP 1909 H=1909:00 +CPU= 26.607
  15:27:19 STEP 1910 H=1910:00 +CPU= 27.134
  15:28:02 STEP 1911 H=1911:00 +CPU= 42.066
  15:28:30 STEP 1912 H=1912:00 +CPU= 26.927
  15:28:58 STEP 1913 H=1913:00 +CPU= 26.025
  15:29:42 STEP 1914 H=1914:00 +CPU= 42.549
  15:30:09 STEP 1915 H=1915:00 +CPU= 26.025
  15:30:36 STEP 1916 H=1916:00 +CPU= 26.911
  15:31:20 STEP 1917 H=1917:00 +CPU= 41.886
  15:31:48 STEP 1918 H=1918:00 +CPU= 27.199
  15:32:16 STEP 1919 H=1919:00 +CPU= 27.435
  15:33:08 STEP 1920 H=1920:00 +CPU= 49.305
Moving to projects directory: /config/slots/8/ICMGGhq0f+001920
Moving to projects directory: /config/slots/8/ICMSHhq0f+001920
Moving to projects directory: /config/slots/8/ICMUAhq0f+001920
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001896
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001896
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001896
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001908
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001908
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001908
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_79.zip
Uploading the intermediate file: upload_file_79.zip
  15:33:35 STEP 1921 H=1921:00 +CPU= 25.968
Uploading trickle at timestep: 6912000
  15:34:03 STEP 1922 H=1922:00 +CPU= 26.860
  15:34:48 STEP 1923 H=1923:00 +CPU= 43.664
  15:35:18 STEP 1924 H=1924:00 +CPU= 28.352
  15:35:46 STEP 1925 H=1925:00 +CPU= 26.871
  15:36:32 STEP 1926 H=1926:00 +CPU= 43.777
  15:37:01 STEP 1927 H=1927:00 +CPU= 27.271
  15:37:30 STEP 1928 H=1928:00 +CPU= 28.587
  15:38:14 STEP 1929 H=1929:00 +CPU= 42.628
  15:38:43 STEP 1930 H=1930:00 +CPU= 28.163
  15:39:12 STEP 1931 H=1931:00 +CPU= 27.407
  15:40:02 STEP 1932 H=1932:00 +CPU= 47.610
Moving to projects directory: /config/slots/8/ICMGGhq0f+001932
Moving to projects directory: /config/slots/8/ICMSHhq0f+001932
Moving to projects directory: /config/slots/8/ICMUAhq0f+001932
  15:40:30 STEP 1933 H=1933:00 +CPU= 27.466
  15:40:59 STEP 1934 H=1934:00 +CPU= 27.800
  15:41:43 STEP 1935 H=1935:00 +CPU= 42.765
  15:42:13 STEP 1936 H=1936:00 +CPU= 28.205
  15:42:40 STEP 1937 H=1937:00 +CPU= 26.964
  15:43:26 STEP 1938 H=1938:00 +CPU= 43.649
  15:43:54 STEP 1939 H=1939:00 +CPU= 27.210
  15:44:24 STEP 1940 H=1940:00 +CPU= 27.922
  15:45:07 STEP 1941 H=1941:00 +CPU= 42.285
  15:45:36 STEP 1942 H=1942:00 +CPU= 27.925
  15:46:06 STEP 1943 H=1943:00 +CPU= 28.256
  15:46:59 STEP 1944 H=1944:00 +CPU= 50.805
Moving to projects directory: /config/slots/8/ICMGGhq0f+001944
Moving to projects directory: /config/slots/8/ICMSHhq0f+001944
Moving to projects directory: /config/slots/8/ICMUAhq0f+001944
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001920
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001920
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001920
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001932
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001932
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001932
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_80.zip
Uploading the intermediate file: upload_file_80.zip
  15:47:28 STEP 1945 H=1945:00 +CPU= 27.501
  15:47:56 STEP 1946 H=1946:00 +CPU= 27.599
  15:48:41 STEP 1947 H=1947:00 +CPU= 42.998
  15:49:11 STEP 1948 H=1948:00 +CPU= 28.163
  15:49:40 STEP 1949 H=1949:00 +CPU= 27.104
  15:50:26 STEP 1950 H=1950:00 +CPU= 43.999
  15:50:54 STEP 1951 H=1951:00 +CPU= 27.657
  15:51:23 STEP 1952 H=1952:00 +CPU= 27.990
  15:52:07 STEP 1953 H=1953:00 +CPU= 42.483
  15:52:37 STEP 1954 H=1954:00 +CPU= 28.544
  15:53:05 STEP 1955 H=1955:00 +CPU= 27.153
  15:53:55 STEP 1956 H=1956:00 +CPU= 47.548
Moving to projects directory: /config/slots/8/ICMGGhq0f+001956
Moving to projects directory: /config/slots/8/ICMSHhq0f+001956
Moving to projects directory: /config/slots/8/ICMUAhq0f+001956
  15:54:24 STEP 1957 H=1957:00 +CPU= 27.440
  15:54:53 STEP 1958 H=1958:00 +CPU= 28.377
  15:55:38 STEP 1959 H=1959:00 +CPU= 43.082
  15:56:08 STEP 1960 H=1960:00 +CPU= 28.346
  15:56:36 STEP 1961 H=1961:00 +CPU= 27.171
  15:57:21 STEP 1962 H=1962:00 +CPU= 44.187
  15:57:50 STEP 1963 H=1963:00 +CPU= 27.321
  15:58:19 STEP 1964 H=1964:00 +CPU= 28.056
  15:59:04 STEP 1965 H=1965:00 +CPU= 43.110
  15:59:33 STEP 1966 H=1966:00 +CPU= 28.203
  16:00:02 STEP 1967 H=1967:00 +CPU= 27.668
  16:00:54 STEP 1968 H=1968:00 +CPU= 50.887
Moving to projects directory: /config/slots/8/ICMGGhq0f+001968
Moving to projects directory: /config/slots/8/ICMSHhq0f+001968
Moving to projects directory: /config/slots/8/ICMUAhq0f+001968
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001944
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001944
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001944
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001956
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001956
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001956
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_81.zip
Uploading the intermediate file: upload_file_81.zip
  16:01:23 STEP 1969 H=1969:00 +CPU= 27.430
  16:01:53 STEP 1970 H=1970:00 +CPU= 27.953
  16:02:38 STEP 1971 H=1971:00 +CPU= 42.703
  16:03:07 STEP 1972 H=1972:00 +CPU= 28.562
  16:03:36 STEP 1973 H=1973:00 +CPU= 27.075
  16:04:21 STEP 1974 H=1974:00 +CPU= 43.737
  16:04:50 STEP 1975 H=1975:00 +CPU= 27.350
  16:05:21 STEP 1976 H=1976:00 +CPU= 28.063
  16:06:05 STEP 1977 H=1977:00 +CPU= 42.584
  16:06:35 STEP 1978 H=1978:00 +CPU= 28.386
  16:07:03 STEP 1979 H=1979:00 +CPU= 27.453
  16:07:53 STEP 1980 H=1980:00 +CPU= 47.656
Moving to projects directory: /config/slots/8/ICMGGhq0f+001980
Moving to projects directory: /config/slots/8/ICMSHhq0f+001980
Moving to projects directory: /config/slots/8/ICMUAhq0f+001980
  16:08:21 STEP 1981 H=1981:00 +CPU= 27.511
  16:08:51 STEP 1982 H=1982:00 +CPU= 28.274
  16:09:36 STEP 1983 H=1983:00 +CPU= 43.141
  16:10:06 STEP 1984 H=1984:00 +CPU= 27.995
  16:10:34 STEP 1985 H=1985:00 +CPU= 27.650
  16:11:20 STEP 1986 H=1986:00 +CPU= 43.923
  16:11:48 STEP 1987 H=1987:00 +CPU= 27.218
  16:12:17 STEP 1988 H=1988:00 +CPU= 27.680
  16:13:02 STEP 1989 H=1989:00 +CPU= 42.981
  16:13:31 STEP 1990 H=1990:00 +CPU= 27.801
  16:14:00 STEP 1991 H=1991:00 +CPU= 28.176
  16:14:52 STEP 1992 H=1992:00 +CPU= 50.720
Moving to projects directory: /config/slots/8/ICMGGhq0f+001992
Moving to projects directory: /config/slots/8/ICMSHhq0f+001992
Moving to projects directory: /config/slots/8/ICMUAhq0f+001992
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001968
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001968
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001968
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001980
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001980
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001980
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_82.zip
Uploading the intermediate file: upload_file_82.zip
  16:15:21 STEP 1993 H=1993:00 +CPU= 27.151
  16:15:50 STEP 1994 H=1994:00 +CPU= 28.091
  16:16:35 STEP 1995 H=1995:00 +CPU= 43.388
  16:17:05 STEP 1996 H=1996:00 +CPU= 28.224
  16:17:33 STEP 1997 H=1997:00 +CPU= 27.233
  16:18:18 STEP 1998 H=1998:00 +CPU= 43.648
  16:18:47 STEP 1999 H=1999:00 +CPU= 27.640
  16:19:17 STEP 2000 H=2000:00 +CPU= 28.545
  16:20:00 STEP 2001 H=2001:00 +CPU= 42.129
  16:20:30 STEP 2002 H=2002:00 +CPU= 28.491
  16:20:58 STEP 2003 H=2003:00 +CPU= 27.274
  16:21:48 STEP 2004 H=2004:00 +CPU= 47.665
Moving to projects directory: /config/slots/8/ICMGGhq0f+002004
Moving to projects directory: /config/slots/8/ICMSHhq0f+002004
Moving to projects directory: /config/slots/8/ICMUAhq0f+002004
  16:22:16 STEP 2005 H=2005:00 +CPU= 27.324
  16:22:47 STEP 2006 H=2006:00 +CPU= 28.472
  16:23:32 STEP 2007 H=2007:00 +CPU= 42.931
  16:24:01 STEP 2008 H=2008:00 +CPU= 28.557
  16:24:31 STEP 2009 H=2009:00 +CPU= 27.984
  16:25:16 STEP 2010 H=2010:00 +CPU= 43.581
  16:25:44 STEP 2011 H=2011:00 +CPU= 27.647
  16:26:14 STEP 2012 H=2012:00 +CPU= 28.249
  16:26:59 STEP 2013 H=2013:00 +CPU= 43.257
  16:27:29 STEP 2014 H=2014:00 +CPU= 28.558
  16:27:59 STEP 2015 H=2015:00 +CPU= 28.905
  16:28:52 STEP 2016 H=2016:00 +CPU= 50.521
Moving to projects directory: /config/slots/8/ICMGGhq0f+002016
Moving to projects directory: /config/slots/8/ICMSHhq0f+002016
Moving to projects directory: /config/slots/8/ICMUAhq0f+002016
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+001992
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+001992
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+001992
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002004
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002004
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002004
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_83.zip
Uploading the intermediate file: upload_file_83.zip
  16:29:20 STEP 2017 H=2017:00 +CPU= 27.378
  16:29:49 STEP 2018 H=2018:00 +CPU= 27.912
  16:30:34 STEP 2019 H=2019:00 +CPU= 43.009
  16:31:03 STEP 2020 H=2020:00 +CPU= 28.186
  16:31:31 STEP 2021 H=2021:00 +CPU= 27.103
  16:32:17 STEP 2022 H=2022:00 +CPU= 44.426
  16:32:46 STEP 2023 H=2023:00 +CPU= 27.847
  16:33:16 STEP 2024 H=2024:00 +CPU= 28.212
  16:34:01 STEP 2025 H=2025:00 +CPU= 43.302
  16:34:30 STEP 2026 H=2026:00 +CPU= 28.079
  16:34:59 STEP 2027 H=2027:00 +CPU= 27.882
  16:35:49 STEP 2028 H=2028:00 +CPU= 47.921
Moving to projects directory: /config/slots/8/ICMGGhq0f+002028
Moving to projects directory: /config/slots/8/ICMSHhq0f+002028
Moving to projects directory: /config/slots/8/ICMUAhq0f+002028
  16:36:17 STEP 2029 H=2029:00 +CPU= 27.385
  16:36:48 STEP 2030 H=2030:00 +CPU= 28.476
  16:37:32 STEP 2031 H=2031:00 +CPU= 42.912
  16:38:02 STEP 2032 H=2032:00 +CPU= 28.247
  16:38:31 STEP 2033 H=2033:00 +CPU= 27.648
  16:39:16 STEP 2034 H=2034:00 +CPU= 43.767
  16:39:45 STEP 2035 H=2035:00 +CPU= 27.611
  16:40:15 STEP 2036 H=2036:00 +CPU= 28.338
  16:41:00 STEP 2037 H=2037:00 +CPU= 43.285
  16:41:29 STEP 2038 H=2038:00 +CPU= 28.216
  16:41:59 STEP 2039 H=2039:00 +CPU= 28.415
  16:42:53 STEP 2040 H=2040:00 +CPU= 50.975
Moving to projects directory: /config/slots/8/ICMGGhq0f+002040
Moving to projects directory: /config/slots/8/ICMSHhq0f+002040
Moving to projects directory: /config/slots/8/ICMUAhq0f+002040
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002016
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002016
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002016
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002028
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002028
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002028
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_84.zip
Uploading the intermediate file: upload_file_84.zip
  16:43:22 STEP 2041 H=2041:00 +CPU= 27.854
  16:43:51 STEP 2042 H=2042:00 +CPU= 28.415
  16:44:36 STEP 2043 H=2043:00 +CPU= 42.487
  16:45:05 STEP 2044 H=2044:00 +CPU= 28.105
  16:45:34 STEP 2045 H=2045:00 +CPU= 27.211
  16:46:19 STEP 2046 H=2046:00 +CPU= 43.807
  16:46:48 STEP 2047 H=2047:00 +CPU= 27.376
  16:47:18 STEP 2048 H=2048:00 +CPU= 28.883
  16:48:03 STEP 2049 H=2049:00 +CPU= 42.960
  16:48:33 STEP 2050 H=2050:00 +CPU= 28.171
  16:49:02 STEP 2051 H=2051:00 +CPU= 27.091
  16:49:53 STEP 2052 H=2052:00 +CPU= 48.383
Moving to projects directory: /config/slots/8/ICMGGhq0f+002052
Moving to projects directory: /config/slots/8/ICMSHhq0f+002052
Moving to projects directory: /config/slots/8/ICMUAhq0f+002052
  16:50:21 STEP 2053 H=2053:00 +CPU= 27.151
  16:50:51 STEP 2054 H=2054:00 +CPU= 28.384
  16:51:36 STEP 2055 H=2055:00 +CPU= 43.137
  16:52:06 STEP 2056 H=2056:00 +CPU= 28.717
  16:52:35 STEP 2057 H=2057:00 +CPU= 27.449
  16:53:21 STEP 2058 H=2058:00 +CPU= 43.918
  16:53:50 STEP 2059 H=2059:00 +CPU= 27.173
  16:54:19 STEP 2060 H=2060:00 +CPU= 28.063
  16:55:04 STEP 2061 H=2061:00 +CPU= 43.441
  16:55:34 STEP 2062 H=2062:00 +CPU= 28.377
  16:56:04 STEP 2063 H=2063:00 +CPU= 28.804
  16:56:57 STEP 2064 H=2064:00 +CPU= 50.862
Moving to projects directory: /config/slots/8/ICMGGhq0f+002064
Moving to projects directory: /config/slots/8/ICMSHhq0f+002064
Moving to projects directory: /config/slots/8/ICMUAhq0f+002064
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002040
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002040
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002040
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002052
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002052
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002052
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_85.zip
Uploading the intermediate file: upload_file_85.zip
  16:57:26 STEP 2065 H=2065:00 +CPU= 27.509
  16:57:55 STEP 2066 H=2066:00 +CPU= 28.078
  16:58:40 STEP 2067 H=2067:00 +CPU= 43.084
  16:59:10 STEP 2068 H=2068:00 +CPU= 28.700
  16:59:38 STEP 2069 H=2069:00 +CPU= 27.268
  17:00:24 STEP 2070 H=2070:00 +CPU= 43.870
  17:00:53 STEP 2071 H=2071:00 +CPU= 27.677
  17:01:22 STEP 2072 H=2072:00 +CPU= 28.131
  17:02:07 STEP 2073 H=2073:00 +CPU= 43.132
  17:02:36 STEP 2074 H=2074:00 +CPU= 28.252
  17:03:06 STEP 2075 H=2075:00 +CPU= 27.678
  17:03:56 STEP 2076 H=2076:00 +CPU= 48.152
Moving to projects directory: /config/slots/8/ICMGGhq0f+002076
Moving to projects directory: /config/slots/8/ICMSHhq0f+002076
Moving to projects directory: /config/slots/8/ICMUAhq0f+002076
  17:04:24 STEP 2077 H=2077:00 +CPU= 27.538
  17:04:54 STEP 2078 H=2078:00 +CPU= 28.524
  17:05:41 STEP 2079 H=2079:00 +CPU= 43.065
  17:06:10 STEP 2080 H=2080:00 +CPU= 28.297
  17:06:39 STEP 2081 H=2081:00 +CPU= 27.409
  17:07:25 STEP 2082 H=2082:00 +CPU= 43.656
  17:07:53 STEP 2083 H=2083:00 +CPU= 27.235
  17:08:22 STEP 2084 H=2084:00 +CPU= 28.366
  17:09:07 STEP 2085 H=2085:00 +CPU= 42.854
  17:09:37 STEP 2086 H=2086:00 +CPU= 28.383
  17:10:07 STEP 2087 H=2087:00 +CPU= 28.765
  17:11:00 STEP 2088 H=2088:00 +CPU= 50.593
Moving to projects directory: /config/slots/8/ICMGGhq0f+002088
Moving to projects directory: /config/slots/8/ICMSHhq0f+002088
Moving to projects directory: /config/slots/8/ICMUAhq0f+002088
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002064
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002064
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002064
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002076
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002076
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002076
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_86.zip
Uploading the intermediate file: upload_file_86.zip
  17:11:29 STEP 2089 H=2089:00 +CPU= 27.379
  17:11:58 STEP 2090 H=2090:00 +CPU= 28.153
  17:12:42 STEP 2091 H=2091:00 +CPU= 42.671
  17:13:11 STEP 2092 H=2092:00 +CPU= 28.030
  17:13:40 STEP 2093 H=2093:00 +CPU= 27.379
  17:14:25 STEP 2094 H=2094:00 +CPU= 43.765
  17:14:53 STEP 2095 H=2095:00 +CPU= 27.286
  17:15:23 STEP 2096 H=2096:00 +CPU= 28.273
  17:16:09 STEP 2097 H=2097:00 +CPU= 43.175
  17:16:38 STEP 2098 H=2098:00 +CPU= 28.249
  17:17:08 STEP 2099 H=2099:00 +CPU= 27.525
  17:17:57 STEP 2100 H=2100:00 +CPU= 47.728
Moving to projects directory: /config/slots/8/ICMGGhq0f+002100
Moving to projects directory: /config/slots/8/ICMSHhq0f+002100
Moving to projects directory: /config/slots/8/ICMUAhq0f+002100
  17:18:26 STEP 2101 H=2101:00 +CPU= 27.354
  17:18:56 STEP 2102 H=2102:00 +CPU= 28.170
  17:19:42 STEP 2103 H=2103:00 +CPU= 42.952
  17:20:12 STEP 2104 H=2104:00 +CPU= 28.111
  17:20:41 STEP 2105 H=2105:00 +CPU= 27.726
  17:21:27 STEP 2106 H=2106:00 +CPU= 43.608
  17:21:56 STEP 2107 H=2107:00 +CPU= 27.329
  17:22:27 STEP 2108 H=2108:00 +CPU= 28.338
  17:23:12 STEP 2109 H=2109:00 +CPU= 42.651
  17:23:41 STEP 2110 H=2110:00 +CPU= 28.219
  17:24:12 STEP 2111 H=2111:00 +CPU= 28.377
  17:25:05 STEP 2112 H=2112:00 +CPU= 50.415
Moving to projects directory: /config/slots/8/ICMGGhq0f+002112
Moving to projects directory: /config/slots/8/ICMSHhq0f+002112
Moving to projects directory: /config/slots/8/ICMUAhq0f+002112
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002088
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002088
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002088
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002100
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002100
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002100
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_87.zip
Uploading the intermediate file: upload_file_87.zip
  17:25:35 STEP 2113 H=2113:00 +CPU= 27.446
  17:26:05 STEP 2114 H=2114:00 +CPU= 28.147
  17:26:51 STEP 2115 H=2115:00 +CPU= 43.542
  17:27:22 STEP 2116 H=2116:00 +CPU= 28.649
  17:27:52 STEP 2117 H=2117:00 +CPU= 27.756
  17:28:39 STEP 2118 H=2118:00 +CPU= 44.934
  17:29:09 STEP 2119 H=2119:00 +CPU= 27.720
  17:29:39 STEP 2120 H=2120:00 +CPU= 28.434
  17:30:26 STEP 2121 H=2121:00 +CPU= 43.690
  17:30:56 STEP 2122 H=2122:00 +CPU= 28.483
  17:31:26 STEP 2123 H=2123:00 +CPU= 27.737
  17:32:16 STEP 2124 H=2124:00 +CPU= 47.862
Moving to projects directory: /config/slots/8/ICMGGhq0f+002124
Moving to projects directory: /config/slots/8/ICMSHhq0f+002124
Moving to projects directory: /config/slots/8/ICMUAhq0f+002124
  17:32:44 STEP 2125 H=2125:00 +CPU= 27.579
  17:33:14 STEP 2126 H=2126:00 +CPU= 28.713
  17:34:00 STEP 2127 H=2127:00 +CPU= 43.695
  17:34:30 STEP 2128 H=2128:00 +CPU= 28.912
  17:34:59 STEP 2129 H=2129:00 +CPU= 27.705
  17:35:45 STEP 2130 H=2130:00 +CPU= 44.652
  17:36:14 STEP 2131 H=2131:00 +CPU= 27.628
  17:36:43 STEP 2132 H=2132:00 +CPU= 28.424
  17:37:29 STEP 2133 H=2133:00 +CPU= 43.512
  17:37:58 STEP 2134 H=2134:00 +CPU= 28.577
  17:38:28 STEP 2135 H=2135:00 +CPU= 28.707
  17:39:22 STEP 2136 H=2136:00 +CPU= 50.990
Moving to projects directory: /config/slots/8/ICMGGhq0f+002136
Moving to projects directory: /config/slots/8/ICMSHhq0f+002136
Moving to projects directory: /config/slots/8/ICMUAhq0f+002136
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002112
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002112
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002112
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002124
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002124
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002124
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_88.zip
Uploading the intermediate file: upload_file_88.zip
  17:39:52 STEP 2137 H=2137:00 +CPU= 27.845
  17:40:23 STEP 2138 H=2138:00 +CPU= 28.302
  17:41:08 STEP 2139 H=2139:00 +CPU= 43.368
  17:41:39 STEP 2140 H=2140:00 +CPU= 28.718
  17:42:09 STEP 2141 H=2141:00 +CPU= 27.743
  17:42:55 STEP 2142 H=2142:00 +CPU= 44.418
  17:43:24 STEP 2143 H=2143:00 +CPU= 27.789
  17:43:54 STEP 2144 H=2144:00 +CPU= 28.894
  17:44:39 STEP 2145 H=2145:00 +CPU= 43.303
  17:45:08 STEP 2146 H=2146:00 +CPU= 28.158
  17:45:37 STEP 2147 H=2147:00 +CPU= 27.729
  17:46:27 STEP 2148 H=2148:00 +CPU= 48.410
Moving to projects directory: /config/slots/8/ICMGGhq0f+002148
Moving to projects directory: /config/slots/8/ICMSHhq0f+002148
Moving to projects directory: /config/slots/8/ICMUAhq0f+002148
  17:46:57 STEP 2149 H=2149:00 +CPU= 27.947
  17:47:27 STEP 2150 H=2150:00 +CPU= 29.117
  17:48:12 STEP 2151 H=2151:00 +CPU= 43.351
  17:48:42 STEP 2152 H=2152:00 +CPU= 28.707
  17:49:11 STEP 2153 H=2153:00 +CPU= 27.658
  17:49:58 STEP 2154 H=2154:00 +CPU= 44.982
  17:50:27 STEP 2155 H=2155:00 +CPU= 27.929
  17:50:58 STEP 2156 H=2156:00 +CPU= 29.377
  17:51:44 STEP 2157 H=2157:00 +CPU= 44.210
  17:52:14 STEP 2158 H=2158:00 +CPU= 28.784
  17:52:44 STEP 2159 H=2159:00 +CPU= 28.922
  17:53:38 STEP 2160 H=2160:00 +CPU= 52.620
Moving to projects directory: /config/slots/8/ICMGGhq0f+002160
Moving to projects directory: /config/slots/8/ICMSHhq0f+002160
Moving to projects directory: /config/slots/8/ICMUAhq0f+002160
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002136
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002136
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002136
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002148
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002148
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002148
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_89.zip
Uploading the intermediate file: upload_file_89.zip
Uploading trickle at timestep: 7776000
  17:54:08 STEP 2161 H=2161:00 +CPU= 28.187
  17:54:38 STEP 2162 H=2162:00 +CPU= 28.953
  17:55:25 STEP 2163 H=2163:00 +CPU= 44.293
  17:55:55 STEP 2164 H=2164:00 +CPU= 28.671
  17:56:25 STEP 2165 H=2165:00 +CPU= 28.208
  17:57:14 STEP 2166 H=2166:00 +CPU= 44.967
  17:57:43 STEP 2167 H=2167:00 +CPU= 27.915
  17:58:14 STEP 2168 H=2168:00 +CPU= 28.873
  17:59:06 STEP 2169 H=2169:00 +CPU= 44.250
  17:59:42 STEP 2170 H=2170:00 +CPU= 29.291
  18:00:17 STEP 2171 H=2171:00 +CPU= 28.247
  18:01:18 STEP 2172 H=2172:00 +CPU= 49.453
Moving to projects directory: /config/slots/8/ICMGGhq0f+002172
Moving to projects directory: /config/slots/8/ICMSHhq0f+002172
Moving to projects directory: /config/slots/8/ICMUAhq0f+002172
  18:01:51 STEP 2173 H=2173:00 +CPU= 28.201
  18:02:21 STEP 2174 H=2174:00 +CPU= 29.188
  18:03:08 STEP 2175 H=2175:00 +CPU= 44.442
  18:03:38 STEP 2176 H=2176:00 +CPU= 28.517
  18:04:07 STEP 2177 H=2177:00 +CPU= 28.255
  18:04:54 STEP 2178 H=2178:00 +CPU= 45.763
  18:05:25 STEP 2179 H=2179:00 +CPU= 28.050
  18:05:55 STEP 2180 H=2180:00 +CPU= 28.494
  18:06:42 STEP 2181 H=2181:00 +CPU= 44.187
  18:07:13 STEP 2182 H=2182:00 +CPU= 29.175
  18:07:43 STEP 2183 H=2183:00 +CPU= 29.058
  18:08:37 STEP 2184 H=2184:00 +CPU= 52.302
Moving to projects directory: /config/slots/8/ICMGGhq0f+002184
Moving to projects directory: /config/slots/8/ICMSHhq0f+002184
Moving to projects directory: /config/slots/8/ICMUAhq0f+002184
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002160
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002160
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002160
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002172
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002172
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002172
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_90.zip
Uploading the intermediate file: upload_file_90.zip
  18:09:06 STEP 2185 H=2185:00 +CPU= 27.579
  18:09:36 STEP 2186 H=2186:00 +CPU= 29.183
  18:10:23 STEP 2187 H=2187:00 +CPU= 44.320
  18:10:54 STEP 2188 H=2188:00 +CPU= 29.309
  18:11:24 STEP 2189 H=2189:00 +CPU= 28.509
  18:12:11 STEP 2190 H=2190:00 +CPU= 44.865
  18:12:40 STEP 2191 H=2191:00 +CPU= 27.787
  18:13:11 STEP 2192 H=2192:00 +CPU= 29.526
  18:13:56 STEP 2193 H=2193:00 +CPU= 43.845
  18:14:26 STEP 2194 H=2194:00 +CPU= 28.877
  18:14:56 STEP 2195 H=2195:00 +CPU= 28.131
  18:16:02 STEP 2196 H=2196:00 +CPU= 50.788
Moving to projects directory: /config/slots/8/ICMGGhq0f+002196
Moving to projects directory: /config/slots/8/ICMSHhq0f+002196
Moving to projects directory: /config/slots/8/ICMUAhq0f+002196
  18:16:48 STEP 2197 H=2197:00 +CPU= 30.328
  18:17:37 STEP 2198 H=2198:00 +CPU= 31.472
  18:18:51 STEP 2199 H=2199:00 +CPU= 47.156
  18:19:38 STEP 2200 H=2200:00 +CPU= 31.392
  18:20:14 STEP 2201 H=2201:00 +CPU= 29.204
  18:21:05 STEP 2202 H=2202:00 +CPU= 45.808
  18:21:52 STEP 2203 H=2203:00 +CPU= 30.351
  18:22:39 STEP 2204 H=2204:00 +CPU= 31.215
  18:23:50 STEP 2205 H=2205:00 +CPU= 46.981
  18:24:39 STEP 2206 H=2206:00 +CPU= 31.147
  18:25:26 STEP 2207 H=2207:00 +CPU= 31.344
  18:26:52 STEP 2208 H=2208:00 +CPU= 55.333
Moving to projects directory: /config/slots/8/ICMGGhq0f+002208
Moving to projects directory: /config/slots/8/ICMSHhq0f+002208
Moving to projects directory: /config/slots/8/ICMUAhq0f+002208
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002184
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002184
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002184
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMGGhq0f+002196
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMSHhq0f+002196
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12205130/ICMUAhq0f+002196
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162_91.zip
Uploading the intermediate file: upload_file_91.zip
  18:27:41 STEP 2209 H=2209:00 +CPU= 30.182
  18:28:29 STEP 2210 H=2210:00 +CPU= 31.395
  18:29:42 STEP 2211 H=2211:00 +CPU= 46.881
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0486
(argv4) batchid: 988
(argv5) wuid: 12205130
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /config/slots/8
Project directory is: /config/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /config/projects/climateprediction.net/oifs_43r3_ps_12205130
..mkdir for temp folder for results failed
Copying: /config/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /config/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /config/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_26213eb517ef4f2283a2ff9f9941f6b1 to: /config/slots/8/oifs_43r3_ps_0486_2019050100_123_988_12205130.zip
Unzipping the namelist zip file: /config/slots/8/oifs_43r3_ps_0486_2019050100_123_988_12205130.zip
ic_ancil_file: ic_ancil_12205130
ifsdata_file: ifsdata_12205130
climate_data_file: clim_data_12205130
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d34402defa1451b07fe2e856b9b8cf9a to: /config/slots/8/ic_ancil_12205130.zip
Unzipping the IC ancils zip file: /config/slots/8/ic_ancil_12205130.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a3917adc88f6f80f0e0a8a1e1bc7b347 to: /config/slots/8/ifsdata/ifsdata_12205130.zip
Unzipping the ifsdata_zip file: /config/slots/8/ifsdata/ifsdata_12205130.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /config/slots/8/159l_2/clim_data_12205130.zip
Unzipping the climate data zip file: /config/slots/8/159l_2/clim_data_12205130.zip
Checking for progress XML file: /config/slots/8/progress_file_12205130.xml
Opened progress file ok : /config/slots/8/progress_file_12205130.xml
-- Model is restarting --
Adjusting last_iter, 2212, to previous model restart step.
Creating progress file: /config/slots/8/progress_file_12205130.xml
last_cpu_time: 70740
upload_file_number: 92
last_iter: 2207
last_upload: 7948800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0486_2019050100_123_988_12205130_2_r826323162
The child process has been launched with process id: 328
Executing the command: /config/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:328:328] [20230208:211913:1675887553:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:21:31 STEP 2208 H=2208:00 +CPU= 43.068
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:45:02 1537759 22304416 oifs_43r3_ps_0486_2019050100_123_988_12205130_2 7,776,000 68,948 0.0089
09 May 2023 16:44:59 1537759 22304416 oifs_43r3_ps_0486_2019050100_123_988_12205130_2 6,912,000 60,892 0.0088
09 May 2023 16:44:56 1537759 22304416 oifs_43r3_ps_0486_2019050100_123_988_12205130_2 6,048,000 53,112 0.0088
09 May 2023 16:44:53 1537759 22304416 oifs_43r3_ps_0486_2019050100_123_988_12205130_2 5,184,000 45,482 0.0088
09 May 2023 16:44:50 1537759 22304416 oifs_43r3_ps_0486_2019050100_123_988_12205130_2 4,320,000 37,891 0.0088
09 May 2023 16:44:47 1537759 22304416 oifs_43r3_ps_0486_2019050100_123_988_12205130_2 3,456,000 30,378 0.0088
09 May 2023 16:44:44 1537759 22304416 oifs_43r3_ps_0486_2019050100_123_988_12205130_2 2,592,000 22,828 0.0088
09 May 2023 16:44:40 1537759 22304416 oifs_43r3_ps_0486_2019050100_123_988_12205130_2 1,728,000 15,269 0.0088
09 May 2023 16:44:36 1537759 22304416 oifs_43r3_ps_0486_2019050100_123_988_12205130_2 864,000 7,697 0.0089


©2024 cpdn.org