climateprediction.net home page
Task 22272416

Task 22272416

Name oifs_43r3_ps_0373_2001050100_123_970_12187017_0
Workunit 12187017
Created 22 Dec 2022, 23:53:13 UTC
Sent 28 Dec 2022, 3:02:43 UTC
Report deadline 27 Jan 2023, 3:02:43 UTC
Received 31 Dec 2022, 18:08:13 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1479204
Run time 1 days 12 hours 46 min 55 sec
CPU time 22 hours 50 min 20 sec
Validate state Invalid
Credit 9,434.88
Device peak FLOPS 3.01 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,417.62 MB
Peak swap size 4,978.09 MB
Peak disk usage 7,614.00 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
017_0_r1580279269_37.zip
Uploading the intermediate file: upload_file_37.zip
  06:28:33 STEP 1609 H=1609:00 +CPU= 52.227
  06:29:30 STEP 1610 H=1610:00 +CPU= 53.779
  06:30:58 STEP 1611 H=1611:00 +CPU= 81.875
  06:31:56 STEP 1612 H=1612:00 +CPU= 54.180
  06:32:53 STEP 1613 H=1613:00 +CPU= 52.788
  06:34:24 STEP 1614 H=1614:00 +CPU= 85.377
  06:35:19 STEP 1615 H=1615:00 +CPU= 51.602
  06:36:16 STEP 1616 H=1616:00 +CPU= 54.784
  06:37:43 STEP 1617 H=1617:00 +CPU= 83.732
  06:38:41 STEP 1618 H=1618:00 +CPU= 54.312
  06:39:38 STEP 1619 H=1619:00 +CPU= 52.802
  06:41:19 STEP 1620 H=1620:00 +CPU= 93.462
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001620
  06:42:16 STEP 1621 H=1621:00 +CPU= 52.383
  06:43:15 STEP 1622 H=1622:00 +CPU= 54.484
  06:44:45 STEP 1623 H=1623:00 +CPU= 84.511
  06:45:45 STEP 1624 H=1624:00 +CPU= 54.931
  06:46:42 STEP 1625 H=1625:00 +CPU= 52.652
  06:48:15 STEP 1626 H=1626:00 +CPU= 86.664
  06:49:12 STEP 1627 H=1627:00 +CPU= 52.775
  06:50:12 STEP 1628 H=1628:00 +CPU= 55.041
  06:51:43 STEP 1629 H=1629:00 +CPU= 84.980
  06:52:42 STEP 1630 H=1630:00 +CPU= 54.869
  06:53:41 STEP 1631 H=1631:00 +CPU= 54.781
  06:55:24 STEP 1632 H=1632:00 +CPU= 99.011
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_38.zip
Uploading the intermediate file: upload_file_38.zip
Uploading trickle at timestep: 5875200
  06:56:18 STEP 1633 H=1633:00 +CPU= 53.032
  06:57:14 STEP 1634 H=1634:00 +CPU= 54.483
  06:58:39 STEP 1635 H=1635:00 +CPU= 83.920
  06:59:36 STEP 1636 H=1636:00 +CPU= 54.995
  07:00:30 STEP 1637 H=1637:00 +CPU= 53.213
  07:01:58 STEP 1638 H=1638:00 +CPU= 86.134
  07:02:52 STEP 1639 H=1639:00 +CPU= 52.996
  07:03:48 STEP 1640 H=1640:00 +CPU= 54.751
  07:05:14 STEP 1641 H=1641:00 +CPU= 84.155
  07:06:10 STEP 1642 H=1642:00 +CPU= 54.644
  07:07:02 STEP 1643 H=1643:00 +CPU= 51.412
  07:08:36 STEP 1644 H=1644:00 +CPU= 92.017
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001644
  07:09:28 STEP 1645 H=1645:00 +CPU= 51.437
  07:10:22 STEP 1646 H=1646:00 +CPU= 53.170
  07:11:46 STEP 1647 H=1647:00 +CPU= 82.990
  07:12:41 STEP 1648 H=1648:00 +CPU= 53.638
  07:13:34 STEP 1649 H=1649:00 +CPU= 51.616
  07:15:01 STEP 1650 H=1650:00 +CPU= 85.474
  07:15:54 STEP 1651 H=1651:00 +CPU= 52.238
  07:16:49 STEP 1652 H=1652:00 +CPU= 53.842
  07:18:14 STEP 1653 H=1653:00 +CPU= 83.464
  07:19:09 STEP 1654 H=1654:00 +CPU= 54.272
  07:20:03 STEP 1655 H=1655:00 +CPU= 53.970
  07:21:42 STEP 1656 H=1656:00 +CPU= 98.429
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_39.zip
Uploading the intermediate file: upload_file_39.zip
  07:22:36 STEP 1657 H=1657:00 +CPU= 52.489
  07:23:30 STEP 1658 H=1658:00 +CPU= 53.929
  07:24:54 STEP 1659 H=1659:00 +CPU= 83.391
  07:25:48 STEP 1660 H=1660:00 +CPU= 52.999
  07:26:41 STEP 1661 H=1661:00 +CPU= 52.219
  07:28:07 STEP 1662 H=1662:00 +CPU= 84.985
  07:28:59 STEP 1663 H=1663:00 +CPU= 52.314
  07:29:54 STEP 1664 H=1664:00 +CPU= 54.376
  07:31:18 STEP 1665 H=1665:00 +CPU= 82.934
  07:32:13 STEP 1666 H=1666:00 +CPU= 54.212
  07:33:06 STEP 1667 H=1667:00 +CPU= 51.857
  07:34:39 STEP 1668 H=1668:00 +CPU= 92.585
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001668
  07:35:34 STEP 1669 H=1669:00 +CPU= 51.494
  07:36:32 STEP 1670 H=1670:00 +CPU= 54.074
  07:38:00 STEP 1671 H=1671:00 +CPU= 83.072
  07:38:56 STEP 1672 H=1672:00 +CPU= 52.855
  07:39:53 STEP 1673 H=1673:00 +CPU= 52.281
  07:41:23 STEP 1674 H=1674:00 +CPU= 84.120
  07:42:17 STEP 1675 H=1675:00 +CPU= 50.959
  07:43:14 STEP 1676 H=1676:00 +CPU= 52.969
  07:44:41 STEP 1677 H=1677:00 +CPU= 82.200
  07:45:38 STEP 1678 H=1678:00 +CPU= 52.987
  07:46:34 STEP 1679 H=1679:00 +CPU= 52.867
  07:48:15 STEP 1680 H=1680:00 +CPU= 96.516
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_40.zip
Uploading the intermediate file: upload_file_40.zip
  07:49:09 STEP 1681 H=1681:00 +CPU= 50.272
  07:50:05 STEP 1682 H=1682:00 +CPU= 52.464
  07:51:34 STEP 1683 H=1683:00 +CPU= 82.711
  07:52:29 STEP 1684 H=1684:00 +CPU= 52.685
  07:53:24 STEP 1685 H=1685:00 +CPU= 51.387
  07:54:55 STEP 1686 H=1686:00 +CPU= 85.164
  07:55:49 STEP 1687 H=1687:00 +CPU= 51.680
  07:56:46 STEP 1688 H=1688:00 +CPU= 53.329
  07:58:14 STEP 1689 H=1689:00 +CPU= 82.308
  07:59:10 STEP 1690 H=1690:00 +CPU= 52.429
  08:00:04 STEP 1691 H=1691:00 +CPU= 50.778
  08:01:41 STEP 1692 H=1692:00 +CPU= 91.758
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001692
  08:02:35 STEP 1693 H=1693:00 +CPU= 50.824
  08:03:32 STEP 1694 H=1694:00 +CPU= 53.466
  08:04:57 STEP 1695 H=1695:00 +CPU= 81.703
  08:05:50 STEP 1696 H=1696:00 +CPU= 52.366
  08:06:41 STEP 1697 H=1697:00 +CPU= 50.069
  08:08:05 STEP 1698 H=1698:00 +CPU= 82.556
  08:08:56 STEP 1699 H=1699:00 +CPU= 50.585
  08:09:48 STEP 1700 H=1700:00 +CPU= 51.580
  08:11:10 STEP 1701 H=1701:00 +CPU= 80.713
  08:12:02 STEP 1702 H=1702:00 +CPU= 51.446
  08:12:54 STEP 1703 H=1703:00 +CPU= 51.162
  08:14:29 STEP 1704 H=1704:00 +CPU= 95.007
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_41.zip
Uploading the intermediate file: upload_file_41.zip
  08:15:20 STEP 1705 H=1705:00 +CPU= 50.202
  08:16:13 STEP 1706 H=1706:00 +CPU= 52.033
  08:17:35 STEP 1707 H=1707:00 +CPU= 81.201
  08:18:27 STEP 1708 H=1708:00 +CPU= 51.944
  08:19:18 STEP 1709 H=1709:00 +CPU= 50.379
  08:20:41 STEP 1710 H=1710:00 +CPU= 82.821
  08:21:32 STEP 1711 H=1711:00 +CPU= 50.346
  08:22:25 STEP 1712 H=1712:00 +CPU= 51.782
  08:23:47 STEP 1713 H=1713:00 +CPU= 81.341
  08:24:39 STEP 1714 H=1714:00 +CPU= 51.646
  08:25:30 STEP 1715 H=1715:00 +CPU= 50.391
  08:27:03 STEP 1716 H=1716:00 +CPU= 92.276
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001716
  08:27:55 STEP 1717 H=1717:00 +CPU= 51.419
  08:28:48 STEP 1718 H=1718:00 +CPU= 52.778
  08:30:11 STEP 1719 H=1719:00 +CPU= 82.384
  08:31:05 STEP 1720 H=1720:00 +CPU= 53.039
  08:31:58 STEP 1721 H=1721:00 +CPU= 51.957
  08:33:22 STEP 1722 H=1722:00 +CPU= 83.397
  08:34:15 STEP 1723 H=1723:00 +CPU= 51.710
  08:35:09 STEP 1724 H=1724:00 +CPU= 53.438
  08:36:33 STEP 1725 H=1725:00 +CPU= 82.990
  08:37:27 STEP 1726 H=1726:00 +CPU= 52.653
  08:38:20 STEP 1727 H=1727:00 +CPU= 52.802
  08:39:58 STEP 1728 H=1728:00 +CPU= 96.660
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_42.zip
Uploading the intermediate file: upload_file_42.zip
  08:40:50 STEP 1729 H=1729:00 +CPU= 51.335
  08:41:44 STEP 1730 H=1730:00 +CPU= 52.556
  08:43:10 STEP 1731 H=1731:00 +CPU= 82.828
  08:44:05 STEP 1732 H=1732:00 +CPU= 53.937
  08:45:00 STEP 1733 H=1733:00 +CPU= 53.340
  08:46:26 STEP 1734 H=1734:00 +CPU= 85.596
  08:47:20 STEP 1735 H=1735:00 +CPU= 52.071
  08:48:15 STEP 1736 H=1736:00 +CPU= 53.872
  08:49:40 STEP 1737 H=1737:00 +CPU= 82.715
  08:50:34 STEP 1738 H=1738:00 +CPU= 52.714
  08:51:27 STEP 1739 H=1739:00 +CPU= 51.903
  08:52:59 STEP 1740 H=1740:00 +CPU= 91.731
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001740
  08:53:51 STEP 1741 H=1741:00 +CPU= 51.480
  08:54:45 STEP 1742 H=1742:00 +CPU= 52.777
  08:56:08 STEP 1743 H=1743:00 +CPU= 82.850
  08:57:02 STEP 1744 H=1744:00 +CPU= 52.615
  08:57:54 STEP 1745 H=1745:00 +CPU= 51.501
  08:59:18 STEP 1746 H=1746:00 +CPU= 83.698
  09:00:10 STEP 1747 H=1747:00 +CPU= 51.492
  09:01:04 STEP 1748 H=1748:00 +CPU= 52.837
  09:02:26 STEP 1749 H=1749:00 +CPU= 81.733
  09:03:20 STEP 1750 H=1750:00 +CPU= 52.807
  09:04:13 STEP 1751 H=1751:00 +CPU= 52.817
  09:05:50 STEP 1752 H=1752:00 +CPU= 96.287
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_43.zip
Uploading the intermediate file: upload_file_43.zip
  09:06:42 STEP 1753 H=1753:00 +CPU= 51.178
  09:07:36 STEP 1754 H=1754:00 +CPU= 53.290
  09:08:59 STEP 1755 H=1755:00 +CPU= 82.187
  09:09:52 STEP 1756 H=1756:00 +CPU= 52.402
  09:10:43 STEP 1757 H=1757:00 +CPU= 51.242
  09:12:09 STEP 1758 H=1758:00 +CPU= 84.685
  09:13:00 STEP 1759 H=1759:00 +CPU= 51.104
  09:13:54 STEP 1760 H=1760:00 +CPU= 52.683
  09:15:18 STEP 1761 H=1761:00 +CPU= 83.018
  09:16:12 STEP 1762 H=1762:00 +CPU= 53.454
  09:17:04 STEP 1763 H=1763:00 +CPU= 52.350
  09:18:36 STEP 1764 H=1764:00 +CPU= 91.209
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001764
  09:19:28 STEP 1765 H=1765:00 +CPU= 51.282
  09:20:21 STEP 1766 H=1766:00 +CPU= 52.770
  09:21:45 STEP 1767 H=1767:00 +CPU= 82.451
  09:22:38 STEP 1768 H=1768:00 +CPU= 52.684
  09:23:31 STEP 1769 H=1769:00 +CPU= 51.924
  09:24:56 STEP 1770 H=1770:00 +CPU= 84.125
  09:25:48 STEP 1771 H=1771:00 +CPU= 51.587
  09:26:42 STEP 1772 H=1772:00 +CPU= 53.746
  09:28:07 STEP 1773 H=1773:00 +CPU= 83.387
  09:29:01 STEP 1774 H=1774:00 +CPU= 53.018
  09:29:54 STEP 1775 H=1775:00 +CPU= 53.248
  09:31:33 STEP 1776 H=1776:00 +CPU= 96.772
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_44.zip
Uploading the intermediate file: upload_file_44.zip
  09:32:25 STEP 1777 H=1777:00 +CPU= 51.273
  09:33:19 STEP 1778 H=1778:00 +CPU= 52.882
  09:34:43 STEP 1779 H=1779:00 +CPU= 83.029
  09:35:36 STEP 1780 H=1780:00 +CPU= 52.890
  09:36:28 STEP 1781 H=1781:00 +CPU= 51.057
  09:37:54 STEP 1782 H=1782:00 +CPU= 84.945
  09:38:46 STEP 1783 H=1783:00 +CPU= 51.685
  09:39:39 STEP 1784 H=1784:00 +CPU= 53.085
  09:41:03 STEP 1785 H=1785:00 +CPU= 82.789
  09:41:57 STEP 1786 H=1786:00 +CPU= 53.128
  09:42:49 STEP 1787 H=1787:00 +CPU= 50.966
  09:44:20 STEP 1788 H=1788:00 +CPU= 90.824
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001788
  09:45:13 STEP 1789 H=1789:00 +CPU= 51.690
  09:46:06 STEP 1790 H=1790:00 +CPU= 52.964
  09:47:30 STEP 1791 H=1791:00 +CPU= 82.701
  09:48:25 STEP 1792 H=1792:00 +CPU= 53.373
  09:49:17 STEP 1793 H=1793:00 +CPU= 51.292
  09:50:41 STEP 1794 H=1794:00 +CPU= 83.507
  09:51:32 STEP 1795 H=1795:00 +CPU= 50.952
  09:52:27 STEP 1796 H=1796:00 +CPU= 53.144
  09:53:51 STEP 1797 H=1797:00 +CPU= 82.171
  09:54:46 STEP 1798 H=1798:00 +CPU= 53.130
  09:55:40 STEP 1799 H=1799:00 +CPU= 53.115
  09:57:18 STEP 1800 H=1800:00 +CPU= 95.820
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_45.zip
Uploading the intermediate file: upload_file_45.zip
  09:58:10 STEP 1801 H=1801:00 +CPU= 51.194
  09:59:03 STEP 1802 H=1802:00 +CPU= 52.255
  10:00:26 STEP 1803 H=1803:00 +CPU= 82.207
  10:01:20 STEP 1804 H=1804:00 +CPU= 52.927
  10:02:11 STEP 1805 H=1805:00 +CPU= 49.933
  10:03:35 STEP 1806 H=1806:00 +CPU= 83.226
  10:04:27 STEP 1807 H=1807:00 +CPU= 51.172
  10:05:20 STEP 1808 H=1808:00 +CPU= 52.086
  10:06:42 STEP 1809 H=1809:00 +CPU= 81.663
  10:07:34 STEP 1810 H=1810:00 +CPU= 51.670
  10:08:26 STEP 1811 H=1811:00 +CPU= 51.348
  10:09:58 STEP 1812 H=1812:00 +CPU= 89.747
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001812
  10:10:49 STEP 1813 H=1813:00 +CPU= 50.248
  10:11:42 STEP 1814 H=1814:00 +CPU= 51.713
  10:13:05 STEP 1815 H=1815:00 +CPU= 80.894
  10:13:59 STEP 1816 H=1816:00 +CPU= 52.612
  10:14:50 STEP 1817 H=1817:00 +CPU= 49.703
  10:16:14 STEP 1818 H=1818:00 +CPU= 82.804
  10:17:05 STEP 1819 H=1819:00 +CPU= 50.273
  10:17:57 STEP 1820 H=1820:00 +CPU= 52.281
  10:19:20 STEP 1821 H=1821:00 +CPU= 81.329
  10:20:12 STEP 1822 H=1822:00 +CPU= 52.022
  10:21:06 STEP 1823 H=1823:00 +CPU= 52.779
  10:22:43 STEP 1824 H=1824:00 +CPU= 95.774
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_46.zip
Uploading the intermediate file: upload_file_46.zip
  10:23:35 STEP 1825 H=1825:00 +CPU= 51.023
  10:24:28 STEP 1826 H=1826:00 +CPU= 53.062
  10:25:51 STEP 1827 H=1827:00 +CPU= 82.146
  10:26:46 STEP 1828 H=1828:00 +CPU= 53.426
  10:27:37 STEP 1829 H=1829:00 +CPU= 50.802
  10:29:04 STEP 1830 H=1830:00 +CPU= 85.059
  10:29:56 STEP 1831 H=1831:00 +CPU= 51.989
  10:30:51 STEP 1832 H=1832:00 +CPU= 53.413
  10:32:14 STEP 1833 H=1833:00 +CPU= 82.707
  10:33:08 STEP 1834 H=1834:00 +CPU= 52.867
  10:34:03 STEP 1835 H=1835:00 +CPU= 52.173
  10:35:38 STEP 1836 H=1836:00 +CPU= 91.928
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001836
  10:36:32 STEP 1837 H=1837:00 +CPU= 52.454
  10:37:27 STEP 1838 H=1838:00 +CPU= 53.848
  10:38:52 STEP 1839 H=1839:00 +CPU= 83.328
  10:39:47 STEP 1840 H=1840:00 +CPU= 53.423
  10:40:40 STEP 1841 H=1841:00 +CPU= 51.480
  10:42:08 STEP 1842 H=1842:00 +CPU= 85.349
  10:43:01 STEP 1843 H=1843:00 +CPU= 51.783
  10:43:55 STEP 1844 H=1844:00 +CPU= 53.039
  10:45:19 STEP 1845 H=1845:00 +CPU= 82.391
  10:46:13 STEP 1846 H=1846:00 +CPU= 52.868
  10:47:07 STEP 1847 H=1847:00 +CPU= 53.022
  10:48:47 STEP 1848 H=1848:00 +CPU= 97.305
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_47.zip
Uploading the intermediate file: upload_file_47.zip
  10:49:39 STEP 1849 H=1849:00 +CPU= 51.516
  10:50:34 STEP 1850 H=1850:00 +CPU= 53.550
  10:51:59 STEP 1851 H=1851:00 +CPU= 82.845
  10:52:57 STEP 1852 H=1852:00 +CPU= 53.849
  10:53:51 STEP 1853 H=1853:00 +CPU= 51.622
  10:55:18 STEP 1854 H=1854:00 +CPU= 84.700
  10:56:11 STEP 1855 H=1855:00 +CPU= 51.329
  10:57:05 STEP 1856 H=1856:00 +CPU= 52.742
  10:58:29 STEP 1857 H=1857:00 +CPU= 82.054
  10:59:23 STEP 1858 H=1858:00 +CPU= 52.571
  11:00:15 STEP 1859 H=1859:00 +CPU= 51.322
  11:01:49 STEP 1860 H=1860:00 +CPU= 91.956
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001860
  11:02:42 STEP 1861 H=1861:00 +CPU= 51.681
  11:03:36 STEP 1862 H=1862:00 +CPU= 53.871
  11:05:02 STEP 1863 H=1863:00 +CPU= 83.217
  11:05:56 STEP 1864 H=1864:00 +CPU= 53.386
  11:06:49 STEP 1865 H=1865:00 +CPU= 51.894
  11:08:17 STEP 1866 H=1866:00 +CPU= 85.761
  11:09:10 STEP 1867 H=1867:00 +CPU= 51.634
  11:10:04 STEP 1868 H=1868:00 +CPU= 53.375
  11:11:29 STEP 1869 H=1869:00 +CPU= 83.297
  11:12:24 STEP 1870 H=1870:00 +CPU= 53.504
  11:13:20 STEP 1871 H=1871:00 +CPU= 54.593
  11:15:00 STEP 1872 H=1872:00 +CPU= 97.891
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_48.zip
Uploading the intermediate file: upload_file_48.zip
Uploading trickle at timestep: 6739200
  11:15:54 STEP 1873 H=1873:00 +CPU= 53.102
  11:16:50 STEP 1874 H=1874:00 +CPU= 54.282
  11:18:15 STEP 1875 H=1875:00 +CPU= 81.903
  11:19:11 STEP 1876 H=1876:00 +CPU= 52.583
  11:20:05 STEP 1877 H=1877:00 +CPU= 50.712
  11:21:35 STEP 1878 H=1878:00 +CPU= 83.618
  11:22:29 STEP 1879 H=1879:00 +CPU= 50.569
  11:23:24 STEP 1880 H=1880:00 +CPU= 51.606
  11:24:50 STEP 1881 H=1881:00 +CPU= 81.054
  11:25:45 STEP 1882 H=1882:00 +CPU= 52.269
  11:26:39 STEP 1883 H=1883:00 +CPU= 50.690
  11:28:15 STEP 1884 H=1884:00 +CPU= 90.370
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001884
  11:29:11 STEP 1885 H=1885:00 +CPU= 53.102
  11:30:06 STEP 1886 H=1886:00 +CPU= 53.484
  11:31:33 STEP 1887 H=1887:00 +CPU= 82.118
  11:32:30 STEP 1888 H=1888:00 +CPU= 52.090
  11:33:25 STEP 1889 H=1889:00 +CPU= 51.399
  11:34:55 STEP 1890 H=1890:00 +CPU= 83.343
  11:35:48 STEP 1891 H=1891:00 +CPU= 50.582
  11:36:45 STEP 1892 H=1892:00 +CPU= 52.765
  11:38:13 STEP 1893 H=1893:00 +CPU= 82.338
  11:39:08 STEP 1894 H=1894:00 +CPU= 52.454
  11:40:04 STEP 1895 H=1895:00 +CPU= 52.767
  11:41:45 STEP 1896 H=1896:00 +CPU= 95.363
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_49.zip
Uploading the intermediate file: upload_file_49.zip
  11:42:37 STEP 1897 H=1897:00 +CPU= 50.106
  11:43:29 STEP 1898 H=1898:00 +CPU= 51.805
  11:44:52 STEP 1899 H=1899:00 +CPU= 80.746
  11:45:45 STEP 1900 H=1900:00 +CPU= 52.009
  11:46:37 STEP 1901 H=1901:00 +CPU= 50.253
  11:48:03 STEP 1902 H=1902:00 +CPU= 83.370
  11:48:54 STEP 1903 H=1903:00 +CPU= 50.288
  11:49:47 STEP 1904 H=1904:00 +CPU= 52.206
  11:51:09 STEP 1905 H=1905:00 +CPU= 80.990
  11:52:01 STEP 1906 H=1906:00 +CPU= 52.004
  11:52:52 STEP 1907 H=1907:00 +CPU= 50.273
  11:54:23 STEP 1908 H=1908:00 +CPU= 89.974
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001908
  11:55:14 STEP 1909 H=1909:00 +CPU= 50.300
  11:56:07 STEP 1910 H=1910:00 +CPU= 52.490
  11:57:29 STEP 1911 H=1911:00 +CPU= 81.240
  11:58:23 STEP 1912 H=1912:00 +CPU= 51.739
  11:59:13 STEP 1913 H=1913:00 +CPU= 49.427
  12:00:37 STEP 1914 H=1914:00 +CPU= 83.286
  12:01:27 STEP 1915 H=1915:00 +CPU= 49.560
  12:02:20 STEP 1916 H=1916:00 +CPU= 52.277
  12:03:42 STEP 1917 H=1917:00 +CPU= 80.703
  12:04:34 STEP 1918 H=1918:00 +CPU= 51.854
  12:05:27 STEP 1919 H=1919:00 +CPU= 52.292
  12:07:06 STEP 1920 H=1920:00 +CPU= 95.877
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_50.zip
Uploading the intermediate file: upload_file_50.zip
  12:07:57 STEP 1921 H=1921:00 +CPU= 50.967
  12:08:50 STEP 1922 H=1922:00 +CPU= 52.117
  12:10:14 STEP 1923 H=1923:00 +CPU= 81.235
  12:11:10 STEP 1924 H=1924:00 +CPU= 52.188
  12:12:02 STEP 1925 H=1925:00 +CPU= 49.829
  12:13:31 STEP 1926 H=1926:00 +CPU= 83.297
  12:14:23 STEP 1927 H=1927:00 +CPU= 49.850
  12:15:17 STEP 1928 H=1928:00 +CPU= 52.095
  12:16:43 STEP 1929 H=1929:00 +CPU= 80.863
  12:17:38 STEP 1930 H=1930:00 +CPU= 51.744
  12:18:31 STEP 1931 H=1931:00 +CPU= 50.076
  12:20:07 STEP 1932 H=1932:00 +CPU= 90.736
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001932
  12:21:01 STEP 1933 H=1933:00 +CPU= 50.415
  12:21:56 STEP 1934 H=1934:00 +CPU= 51.986
  12:23:22 STEP 1935 H=1935:00 +CPU= 81.310
  12:24:17 STEP 1936 H=1936:00 +CPU= 52.003
  12:25:11 STEP 1937 H=1937:00 +CPU= 50.425
  12:26:38 STEP 1938 H=1938:00 +CPU= 82.365
  12:27:30 STEP 1939 H=1939:00 +CPU= 49.569
  12:28:25 STEP 1940 H=1940:00 +CPU= 51.714
  12:29:50 STEP 1941 H=1941:00 +CPU= 80.922
  12:30:45 STEP 1942 H=1942:00 +CPU= 52.448
  12:31:40 STEP 1943 H=1943:00 +CPU= 51.649
  12:33:18 STEP 1944 H=1944:00 +CPU= 95.873
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269_51.zip
Uploading the intermediate file: upload_file_51.zip
  12:34:11 STEP 1945 H=1945:00 +CPU= 52.267
  12:35:06 STEP 1946 H=1946:00 +CPU= 54.248
  12:36:30 STEP 1947 H=1947:00 +CPU= 82.954
  12:37:25 STEP 1948 H=1948:00 +CPU= 53.315
  12:38:17 STEP 1949 H=1949:00 +CPU= 51.803
  12:39:40 STEP 1950 H=1950:00 +CPU= 82.245
  12:40:31 STEP 1951 H=1951:00 +CPU= 50.978
  12:41:24 STEP 1952 H=1952:00 +CPU= 52.008
  12:42:51 STEP 1953 H=1953:00 +CPU= 84.931
  12:43:46 STEP 1954 H=1954:00 +CPU= 54.455
  12:44:38 STEP 1955 H=1955:00 +CPU= 51.402
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0373
(argv4) batchid: 970
(argv5) wuid: 12187017
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/19
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_941b9be10e07936b6925e01a492d366f to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0373_2001050100_123_970_12187017.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0373_2001050100_123_970_12187017.zip
ic_ancil_file: ic_ancil_12187017
ifsdata_file: ifsdata_12187017
climate_data_file: clim_data_12187017
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0145cebe58acd4ae715703032ba35e11 to: /var/lib/boinc-client/slots/19/ic_ancil_12187017.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/19/ic_ancil_12187017.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c653cb178161da66d6ceeb3276b12219 to: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12187017.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12187017.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/19/159l_2/clim_data_12187017.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/19/159l_2/clim_data_12187017.zip
Checking for progress XML file: /var/lib/boinc-client/slots/19/progress_file_12187017.xml
Opened progress file ok : /var/lib/boinc-client/slots/19/progress_file_12187017.xml
-- Model is restarting --
Adjusting last_iter, 1956, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/19/progress_file_12187017.xml
last_cpu_time: 80346
upload_file_number: 52
last_iter: 1943
last_upload: 6998400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269
The child process has been launched with process id: 4109
Executing the command: /var/lib/boinc-client/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:4109:4109] [20221231:100743:1672502863:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0373
(argv4) batchid: 970
(argv5) wuid: 12187017
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/19
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_941b9be10e07936b6925e01a492d366f to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0373_2001050100_123_970_12187017.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0373_2001050100_123_970_12187017.zip
ic_ancil_file: ic_ancil_12187017
ifsdata_file: ifsdata_12187017
climate_data_file: clim_data_12187017
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0145cebe58acd4ae715703032ba35e11 to: /var/lib/boinc-client/slots/19/ic_ancil_12187017.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/19/ic_ancil_12187017.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c653cb178161da66d6ceeb3276b12219 to: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12187017.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12187017.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/19/159l_2/clim_data_12187017.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/19/159l_2/clim_data_12187017.zip
Checking for progress XML file: /var/lib/boinc-client/slots/19/progress_file_12187017.xml
Opened progress file ok : /var/lib/boinc-client/slots/19/progress_file_12187017.xml
-- Model is restarting --
Adjusting last_iter, 1944, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/19/progress_file_12187017.xml
last_cpu_time: 80440
upload_file_number: 52
last_iter: 1943
last_upload: 6998400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269
The child process has been launched with process id: 7889
Executing the command: /var/lib/boinc-client/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:7889:7889] [20221231:102157:1672503717:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:23:49 STEP 1944 H=1944:00 +CPU= 88.290
  10:24:42 STEP 1945 H=1945:00 +CPU= 52.439
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0373
(argv4) batchid: 970
(argv5) wuid: 12187017
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/19
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187017
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_941b9be10e07936b6925e01a492d366f to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0373_2001050100_123_970_12187017.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0373_2001050100_123_970_12187017.zip
ic_ancil_file: ic_ancil_12187017
ifsdata_file: ifsdata_12187017
climate_data_file: clim_data_12187017
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0145cebe58acd4ae715703032ba35e11 to: /var/lib/boinc-client/slots/19/ic_ancil_12187017.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/19/ic_ancil_12187017.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c653cb178161da66d6ceeb3276b12219 to: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12187017.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12187017.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/19/159l_2/clim_data_12187017.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/19/159l_2/clim_data_12187017.zip
Checking for progress XML file: /var/lib/boinc-client/slots/19/progress_file_12187017.xml
Opened progress file ok : /var/lib/boinc-client/slots/19/progress_file_12187017.xml
-- Model is restarting --
Adjusting last_iter, 1946, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/19/progress_file_12187017.xml
last_cpu_time: 80601
upload_file_number: 52
last_iter: 1943
last_upload: 6998400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0373_2001050100_123_970_12187017_0_r1580279269
The child process has been launched with process id: 10614
Executing the command: /var/lib/boinc-client/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dionysus:1:1:10614:10614] [20221231:103901:1672504741:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:40:40 STEP 1944 H=1944:00 +CPU= 92.553
  10:41:33 STEP 1945 H=1945:00 +CPU= 52.032
  10:42:27 STEP 1946 H=1946:00 +CPU= 53.181
  10:43:53 STEP 1947 H=1947:00 +CPU= 83.805
  10:44:50 STEP 1948 H=1948:00 +CPU= 56.455
  10:45:45 STEP 1949 H=1949:00 +CPU= 53.380
  10:47:16 STEP 1950 H=1950:00 +CPU= 84.138
  10:48:12 STEP 1951 H=1951:00 +CPU= 51.907
  10:49:12 STEP 1952 H=1952:00 +CPU= 54.087
  10:50:42 STEP 1953 H=1953:00 +CPU= 83.435
  10:51:41 STEP 1954 H=1954:00 +CPU= 54.013
  10:52:37 STEP 1955 H=1955:00 +CPU= 51.770
  10:54:20 STEP 1956 H=1956:00 +CPU= 92.207
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001956
  10:55:17 STEP 1957 H=1957:00 +CPU= 52.070
  10:56:16 STEP 1958 H=1958:00 +CPU= 53.984
  10:57:46 STEP 1959 H=1959:00 +CPU= 83.583
  10:58:43 STEP 1960 H=1960:00 +CPU= 55.851
  10:59:36 STEP 1961 H=1961:00 +CPU= 52.179
  11:00:58 STEP 1962 H=1962:00 +CPU= 80.860
  11:01:48 STEP 1963 H=1963:00 +CPU= 49.269
  11:02:39 STEP 1964 H=1964:00 +CPU= 50.785
  11:03:58 STEP 1965 H=1965:00 +CPU= 77.901
  11:04:48 STEP 1966 H=1966:00 +CPU= 49.837
  11:05:38 STEP 1967 H=1967:00 +CPU= 49.693
  11:07:16 STEP 1968 H=1968:00 +CPU= 94.542
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:01:50 1479204 22272416 oifs_43r3_ps_0373_2001050100_123_970_12187017_0 6,739,200 75,091 0.0111
09 May 2023 16:01:50 1479204 22272416 oifs_43r3_ps_0373_2001050100_123_970_12187017_0 6,739,200 75,091 0.0111
09 May 2023 16:01:33 1479204 22272416 oifs_43r3_ps_0373_2001050100_123_970_12187017_0 5,875,200 59,848 0.0102
09 May 2023 16:01:17 1479204 22272416 oifs_43r3_ps_0373_2001050100_123_970_12187017_0 5,011,200 44,914 0.0090
09 May 2023 16:00:01 1479204 22272416 oifs_43r3_ps_0373_2001050100_123_970_12187017_0 1,728,000 29,556 0.0171
09 May 2023 16:00:01 1479204 22272416 oifs_43r3_ps_0373_2001050100_123_970_12187017_0 1,728,000 29,556 0.0171
09 May 2023 15:59:49 1479204 22272416 oifs_43r3_ps_0373_2001050100_123_970_12187017_0 864,000 14,896 0.0172


©2024 cpdn.org