Name | oifs_43r3_ps_0258_1998050100_123_967_12183902_0 |
Workunit | 12183902 |
Created | 22 Dec 2022, 15:44:09 UTC |
Sent | 27 Dec 2022, 7:10:51 UTC |
Report deadline | 26 Jan 2023, 7:10:51 UTC |
Received | 31 Dec 2022, 20:38:09 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1533721 |
Run time | 8 hours 52 min 21 sec |
CPU time | 7 hours 1 min 10 sec |
Validate state | Invalid |
Credit | 7,862.40 |
Device peak FLOPS | 7.61 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,629.85 MB |
Peak swap size | 4,978.11 MB |
Peak disk usage | 7,237.50 MB |
Stderr | <core_client_version>7.20.4</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> he zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001632 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001632 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001644 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001644 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001644 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_68.zip Uploading the intermediate file: upload_file_68.zip 22:18:08 STEP 1657 H=1657:00 +CPU= 10.603 22:18:22 STEP 1658 H=1658:00 +CPU= 11.835 Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:18:50 STEP 1659 H=1659:00 +CPU= 16.568 22:19:01 STEP 1660 H=1660:00 +CPU= 10.576 22:19:12 STEP 1661 H=1661:00 +CPU= 10.762 22:19:29 STEP 1662 H=1662:00 +CPU= 16.850 22:19:39 STEP 1663 H=1663:00 +CPU= 10.478 22:19:50 STEP 1664 H=1664:00 +CPU= 11.241 22:20:07 STEP 1665 H=1665:00 +CPU= 16.982 22:20:18 STEP 1666 H=1666:00 +CPU= 10.941 22:20:29 STEP 1667 H=1667:00 +CPU= 10.518 22:20:49 STEP 1668 H=1668:00 +CPU= 20.162 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001668 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001668 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001668 22:21:00 STEP 1669 H=1669:00 +CPU= 10.751 22:21:11 STEP 1670 H=1670:00 +CPU= 11.021 22:21:28 STEP 1671 H=1671:00 +CPU= 17.011 22:21:39 STEP 1672 H=1672:00 +CPU= 11.006 22:21:50 STEP 1673 H=1673:00 +CPU= 10.525 22:22:07 STEP 1674 H=1674:00 +CPU= 17.263 22:22:18 STEP 1675 H=1675:00 +CPU= 10.768 22:22:29 STEP 1676 H=1676:00 +CPU= 11.330 22:22:46 STEP 1677 H=1677:00 +CPU= 16.841 22:22:57 STEP 1678 H=1678:00 +CPU= 10.996 22:23:11 STEP 1679 H=1679:00 +CPU= 11.678 Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:23:44 STEP 1680 H=1680:00 +CPU= 19.764 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001680 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001680 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001680 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001656 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001656 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001656 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001668 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001668 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001668 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_69.zip 22:23:54 STEP 1681 H=1681:00 +CPU= 10.146 Uploading the intermediate file: upload_file_69.zip 22:24:05 STEP 1682 H=1682:00 +CPU= 11.063 22:24:22 STEP 1683 H=1683:00 +CPU= 16.395 22:24:33 STEP 1684 H=1684:00 +CPU= 10.836 22:24:43 STEP 1685 H=1685:00 +CPU= 10.788 22:25:01 STEP 1686 H=1686:00 +CPU= 18.010 22:25:12 STEP 1687 H=1687:00 +CPU= 10.542 22:25:23 STEP 1688 H=1688:00 +CPU= 11.241 22:25:40 STEP 1689 H=1689:00 +CPU= 16.781 22:25:51 STEP 1690 H=1690:00 +CPU= 10.926 22:26:02 STEP 1691 H=1691:00 +CPU= 10.460 22:26:20 STEP 1692 H=1692:00 +CPU= 18.313 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001692 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001692 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001692 22:26:30 STEP 1693 H=1693:00 +CPU= 10.260 22:26:41 STEP 1694 H=1694:00 +CPU= 11.010 22:26:58 STEP 1695 H=1695:00 +CPU= 16.490 22:27:09 STEP 1696 H=1696:00 +CPU= 11.039 22:27:20 STEP 1697 H=1697:00 +CPU= 10.704 22:27:37 STEP 1698 H=1698:00 +CPU= 16.863 22:27:47 STEP 1699 H=1699:00 +CPU= 10.551 22:27:58 STEP 1700 H=1700:00 +CPU= 10.965 22:28:15 STEP 1701 H=1701:00 +CPU= 16.723 22:28:26 STEP 1702 H=1702:00 +CPU= 11.294 Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:28:53 STEP 1703 H=1703:00 +CPU= 11.856 22:29:12 STEP 1704 H=1704:00 +CPU= 19.316 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001704 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001704 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001704 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001680 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001680 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001680 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001692 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001692 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001692 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_70.zip Uploading the intermediate file: upload_file_70.zip 22:29:23 STEP 1705 H=1705:00 +CPU= 10.464 22:29:34 STEP 1706 H=1706:00 +CPU= 11.267 22:29:50 STEP 1707 H=1707:00 +CPU= 16.451 22:30:01 STEP 1708 H=1708:00 +CPU= 10.898 22:30:12 STEP 1709 H=1709:00 +CPU= 10.694 22:30:29 STEP 1710 H=1710:00 +CPU= 17.101 22:30:40 STEP 1711 H=1711:00 +CPU= 10.870 22:30:51 STEP 1712 H=1712:00 +CPU= 11.062 22:31:08 STEP 1713 H=1713:00 +CPU= 16.737 22:31:20 STEP 1714 H=1714:00 +CPU= 11.541 22:31:30 STEP 1715 H=1715:00 +CPU= 10.638 22:31:49 STEP 1716 H=1716:00 +CPU= 18.503 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001716 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001716 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001716 22:31:59 STEP 1717 H=1717:00 +CPU= 10.552 22:32:11 STEP 1718 H=1718:00 +CPU= 11.096 22:32:29 STEP 1719 H=1719:00 +CPU= 18.768 22:32:40 STEP 1720 H=1720:00 +CPU= 10.921 22:32:51 STEP 1721 H=1721:00 +CPU= 10.810 22:33:08 STEP 1722 H=1722:00 +CPU= 17.041 22:33:19 STEP 1723 H=1723:00 +CPU= 10.692 22:33:30 STEP 1724 H=1724:00 +CPU= 10.948 22:33:47 STEP 1725 H=1725:00 +CPU= 16.925 22:33:58 STEP 1726 H=1726:00 +CPU= 11.094 22:34:11 STEP 1727 H=1727:00 +CPU= 11.719 Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:34:45 STEP 1728 H=1728:00 +CPU= 20.632 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001728 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001728 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001728 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001704 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001704 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001704 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001716 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001716 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001716 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_71.zip Uploading the intermediate file: upload_file_71.zip 22:34:56 STEP 1729 H=1729:00 +CPU= 10.555 22:35:09 STEP 1730 H=1730:00 +CPU= 11.375 22:35:27 STEP 1731 H=1731:00 +CPU= 16.909 22:35:38 STEP 1732 H=1732:00 +CPU= 10.885 22:35:48 STEP 1733 H=1733:00 +CPU= 10.733 22:36:05 STEP 1734 H=1734:00 +CPU= 16.867 22:36:16 STEP 1735 H=1735:00 +CPU= 10.585 22:36:27 STEP 1736 H=1736:00 +CPU= 10.846 22:36:43 STEP 1737 H=1737:00 +CPU= 15.834 22:36:53 STEP 1738 H=1738:00 +CPU= 10.442 22:37:04 STEP 1739 H=1739:00 +CPU= 10.717 22:37:23 STEP 1740 H=1740:00 +CPU= 18.719 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001740 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001740 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001740 22:37:33 STEP 1741 H=1741:00 +CPU= 10.665 22:37:45 STEP 1742 H=1742:00 +CPU= 11.173 22:38:01 STEP 1743 H=1743:00 +CPU= 16.498 22:38:12 STEP 1744 H=1744:00 +CPU= 11.058 22:38:23 STEP 1745 H=1745:00 +CPU= 10.701 22:38:40 STEP 1746 H=1746:00 +CPU= 17.187 22:38:51 STEP 1747 H=1747:00 +CPU= 10.566 22:39:02 STEP 1748 H=1748:00 +CPU= 10.879 22:39:19 STEP 1749 H=1749:00 +CPU= 16.887 22:39:30 STEP 1750 H=1750:00 +CPU= 10.907 22:39:40 STEP 1751 H=1751:00 +CPU= 10.733 22:40:01 STEP 1752 H=1752:00 +CPU= 20.380 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001752 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001752 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001752 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001728 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001728 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001728 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001740 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001740 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001740 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_72.zip Uploading the intermediate file: upload_file_72.zip 22:40:14 STEP 1753 H=1753:00 +CPU= 13.362 22:40:26 STEP 1754 H=1754:00 +CPU= 11.183 22:40:42 STEP 1755 H=1755:00 +CPU= 16.531 22:40:53 STEP 1756 H=1756:00 +CPU= 11.077 22:41:04 STEP 1757 H=1757:00 +CPU= 10.443 22:41:21 STEP 1758 H=1758:00 +CPU= 16.937 22:41:31 STEP 1759 H=1759:00 +CPU= 10.503 22:41:43 STEP 1760 H=1760:00 +CPU= 11.193 22:41:59 STEP 1761 H=1761:00 +CPU= 16.869 22:42:10 STEP 1762 H=1762:00 +CPU= 10.845 22:42:22 STEP 1763 H=1763:00 +CPU= 11.742 22:42:46 STEP 1764 H=1764:00 +CPU= 24.141 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001764 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001764 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001764 22:42:59 STEP 1765 H=1765:00 +CPU= 12.104 22:43:10 STEP 1766 H=1766:00 +CPU= 11.093 22:43:26 STEP 1767 H=1767:00 +CPU= 16.355 22:43:37 STEP 1768 H=1768:00 +CPU= 10.568 22:43:47 STEP 1769 H=1769:00 +CPU= 10.741 22:44:04 STEP 1770 H=1770:00 +CPU= 16.836 22:44:15 STEP 1771 H=1771:00 +CPU= 10.896 22:44:26 STEP 1772 H=1772:00 +CPU= 11.033 22:44:43 STEP 1773 H=1773:00 +CPU= 16.314 22:44:57 STEP 1774 H=1774:00 +CPU= 13.938 22:45:12 STEP 1775 H=1775:00 +CPU= 15.759 22:45:32 STEP 1776 H=1776:00 +CPU= 19.556 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001776 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001776 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001776 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001752 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001752 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001752 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001764 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001764 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001764 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_73.zip Uploading the intermediate file: upload_file_73.zip 22:45:43 STEP 1777 H=1777:00 +CPU= 10.721 22:45:54 STEP 1778 H=1778:00 +CPU= 10.898 22:46:11 STEP 1779 H=1779:00 +CPU= 16.668 22:46:22 STEP 1780 H=1780:00 +CPU= 10.993 22:46:33 STEP 1781 H=1781:00 +CPU= 10.869 22:46:50 STEP 1782 H=1782:00 +CPU= 17.055 22:47:00 STEP 1783 H=1783:00 +CPU= 10.454 22:47:13 STEP 1784 H=1784:00 +CPU= 10.985 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1998050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0258 (argv4) batchid: 967 (argv5) wuid: 12183902 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/0 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_9f1538260466a9193194173f9ed56c34 to: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip Unzipping the namelist zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip ic_ancil_file: ic_ancil_12183902 ifsdata_file: ifsdata_12183902 climate_data_file: clim_data_12183902 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b3b72de1fea5937ea6fedf6e21b4e9ba to: /var/lib/boinc/slots/0/ic_ancil_12183902.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/0/ic_ancil_12183902.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4f0e1258061265fd68fcb5e19d1d2808 to: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip Unzipping the climate data zip file: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip Checking for progress XML file: /var/lib/boinc/slots/0/progress_file_12183902.xml Opened progress file ok : /var/lib/boinc/slots/0/progress_file_12183902.xml -- Model is restarting -- Adjusting last_iter, 1785, to previous model restart step. Creating progress file: /var/lib/boinc/slots/0/progress_file_12183902.xml last_cpu_time: 22180 upload_file_number: 74 last_iter: 1775 last_upload: 6393600 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715 The child process has been launched with process id: 3409268 Executing the command: /var/lib/boinc/slots/0/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:52:03 STEP 1776 H=1776:00 +CPU= 21.190 22:53:32 STEP 1777 H=1777:00 +CPU= 10.348 22:53:43 STEP 1778 H=1778:00 +CPU= 10.243 22:54:14 STEP 1779 H=1779:00 +CPU= 16.413 22:55:40 STEP 1780 H=1780:00 +CPU= 11.727 22:57:45 STEP 1781 H=1781:00 +CPU= 11.246 22:59:51 STEP 1782 H=1782:00 +CPU= 17.954 23:00:06 STEP 1783 H=1783:00 +CPU= 10.789 23:01:21 STEP 1784 H=1784:00 +CPU= 10.952 23:01:39 STEP 1785 H=1785:00 +CPU= 16.662 23:06:51 STEP 1786 H=1786:00 +CPU= 12.006 23:07:50 STEP 1787 H=1787:00 +CPU= 10.179 23:08:18 STEP 1788 H=1788:00 +CPU= 17.916 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001788 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001788 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001788 23:08:29 STEP 1789 H=1789:00 +CPU= 10.852 23:08:40 STEP 1790 H=1790:00 +CPU= 11.119 23:11:19 STEP 1791 H=1791:00 +CPU= 16.961 23:12:28 STEP 1792 H=1792:00 +CPU= 10.893 23:12:40 STEP 1793 H=1793:00 +CPU= 10.612 23:12:57 STEP 1794 H=1794:00 +CPU= 17.107 23:13:08 STEP 1795 H=1795:00 +CPU= 10.784 23:13:20 STEP 1796 H=1796:00 +CPU= 11.361 23:13:36 STEP 1797 H=1797:00 +CPU= 16.735 23:13:48 STEP 1798 H=1798:00 +CPU= 11.223 23:13:59 STEP 1799 H=1799:00 +CPU= 11.182 23:14:26 STEP 1800 H=1800:00 +CPU= 20.093 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001800 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001800 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001800 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001776 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001776 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001776 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001788 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001788 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001788 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_74.zip Uploading the intermediate file: upload_file_74.zip 23:14:38 STEP 1801 H=1801:00 +CPU= 11.128 23:14:49 STEP 1802 H=1802:00 +CPU= 11.072 23:15:06 STEP 1803 H=1803:00 +CPU= 16.948 23:15:17 STEP 1804 H=1804:00 +CPU= 11.201 23:15:28 STEP 1805 H=1805:00 +CPU= 11.017 23:15:46 STEP 1806 H=1806:00 +CPU= 17.252 23:15:57 STEP 1807 H=1807:00 +CPU= 10.924 23:16:08 STEP 1808 H=1808:00 +CPU= 11.209 23:16:25 STEP 1809 H=1809:00 +CPU= 16.930 23:16:36 STEP 1810 H=1810:00 +CPU= 11.403 23:16:56 STEP 1811 H=1811:00 +CPU= 10.238 23:17:19 STEP 1812 H=1812:00 +CPU= 16.990 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001812 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001812 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001812 23:17:47 STEP 1813 H=1813:00 +CPU= 10.450 23:17:59 STEP 1814 H=1814:00 +CPU= 11.148 23:19:54 STEP 1815 H=1815:00 +CPU= 17.045 23:20:05 STEP 1816 H=1816:00 +CPU= 11.361 23:21:54 STEP 1817 H=1817:00 +CPU= 10.268 23:22:13 STEP 1818 H=1818:00 +CPU= 16.669 23:22:24 STEP 1819 H=1819:00 +CPU= 10.760 23:23:09 STEP 1820 H=1820:00 +CPU= 10.352 23:23:30 STEP 1821 H=1821:00 +CPU= 15.900 23:23:41 STEP 1822 H=1822:00 +CPU= 10.308 23:25:07 STEP 1823 H=1823:00 +CPU= 10.525 23:27:12 STEP 1824 H=1824:00 +CPU= 19.546 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001824 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001824 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001824 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001800 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001800 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001800 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001812 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001812 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001812 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_75.zip Uploading the intermediate file: upload_file_75.zip 23:27:23 STEP 1825 H=1825:00 +CPU= 10.813 23:27:42 STEP 1826 H=1826:00 +CPU= 11.354 23:27:58 STEP 1827 H=1827:00 +CPU= 15.947 23:28:09 STEP 1828 H=1828:00 +CPU= 10.576 23:29:45 STEP 1829 H=1829:00 +CPU= 10.700 23:31:14 STEP 1830 H=1830:00 +CPU= 17.083 23:31:26 STEP 1831 H=1831:00 +CPU= 10.819 23:31:37 STEP 1832 H=1832:00 +CPU= 10.986 23:32:01 STEP 1833 H=1833:00 +CPU= 16.322 23:32:12 STEP 1834 H=1834:00 +CPU= 11.277 23:32:22 STEP 1835 H=1835:00 +CPU= 10.262 23:32:44 STEP 1836 H=1836:00 +CPU= 18.238 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001836 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001836 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001836 23:32:54 STEP 1837 H=1837:00 +CPU= 10.753 23:33:26 STEP 1838 H=1838:00 +CPU= 10.347 23:33:43 STEP 1839 H=1839:00 +CPU= 14.897 23:33:53 STEP 1840 H=1840:00 +CPU= 9.856 23:34:03 STEP 1841 H=1841:00 +CPU= 9.529 23:34:18 STEP 1842 H=1842:00 +CPU= 15.328 23:34:29 STEP 1843 H=1843:00 +CPU= 10.393 23:34:39 STEP 1844 H=1844:00 +CPU= 10.735 23:34:57 STEP 1845 H=1845:00 +CPU= 17.037 23:35:13 STEP 1846 H=1846:00 +CPU= 10.355 23:35:26 STEP 1847 H=1847:00 +CPU= 9.806 23:36:34 STEP 1848 H=1848:00 +CPU= 17.902 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001848 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001848 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001848 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001824 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001824 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001824 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001836 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001836 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001836 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_76.zip Uploading the intermediate file: upload_file_76.zip 23:36:45 STEP 1849 H=1849:00 +CPU= 10.289 23:37:26 STEP 1850 H=1850:00 +CPU= 10.783 23:38:04 STEP 1851 H=1851:00 +CPU= 15.914 23:38:14 STEP 1852 H=1852:00 +CPU= 9.784 23:38:24 STEP 1853 H=1853:00 +CPU= 9.525 23:38:39 STEP 1854 H=1854:00 +CPU= 15.335 23:38:49 STEP 1855 H=1855:00 +CPU= 10.302 23:39:00 STEP 1856 H=1856:00 +CPU= 10.757 23:39:17 STEP 1857 H=1857:00 +CPU= 17.057 23:39:43 STEP 1858 H=1858:00 +CPU= 10.431 23:39:53 STEP 1859 H=1859:00 +CPU= 9.783 23:40:25 STEP 1860 H=1860:00 +CPU= 17.231 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001860 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001860 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001860 23:40:35 STEP 1861 H=1861:00 +CPU= 9.853 23:40:45 STEP 1862 H=1862:00 +CPU= 9.856 23:41:01 STEP 1863 H=1863:00 +CPU= 16.094 23:41:12 STEP 1864 H=1864:00 +CPU= 10.783 23:41:23 STEP 1865 H=1865:00 +CPU= 10.706 23:44:07 STEP 1866 H=1866:00 +CPU= 18.551 23:45:10 STEP 1867 H=1867:00 +CPU= 11.142 23:45:29 STEP 1868 H=1868:00 +CPU= 10.884 23:46:14 STEP 1869 H=1869:00 +CPU= 16.532 23:46:34 STEP 1870 H=1870:00 +CPU= 10.948 23:46:45 STEP 1871 H=1871:00 +CPU= 11.137 23:47:26 STEP 1872 H=1872:00 +CPU= 19.931 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001872 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001872 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001872 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001848 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001848 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001848 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001860 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001860 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001860 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_77.zip Uploading the intermediate file: upload_file_77.zip 23:47:37 STEP 1873 H=1873:00 +CPU= 10.908 23:47:48 STEP 1874 H=1874:00 +CPU= 11.166 23:48:05 STEP 1875 H=1875:00 +CPU= 16.801 23:48:16 STEP 1876 H=1876:00 +CPU= 11.166 23:48:27 STEP 1877 H=1877:00 +CPU= 10.821 23:48:44 STEP 1878 H=1878:00 +CPU= 17.114 23:48:55 STEP 1879 H=1879:00 +CPU= 10.855 23:49:06 STEP 1880 H=1880:00 +CPU= 11.122 23:49:23 STEP 1881 H=1881:00 +CPU= 16.683 23:49:34 STEP 1882 H=1882:00 +CPU= 11.140 23:49:45 STEP 1883 H=1883:00 +CPU= 10.866 23:50:05 STEP 1884 H=1884:00 +CPU= 18.435 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001884 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001884 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001884 23:50:16 STEP 1885 H=1885:00 +CPU= 10.875 23:50:27 STEP 1886 H=1886:00 +CPU= 11.162 23:50:44 STEP 1887 H=1887:00 +CPU= 16.748 23:50:55 STEP 1888 H=1888:00 +CPU= 11.106 23:51:06 STEP 1889 H=1889:00 +CPU= 10.788 23:51:23 STEP 1890 H=1890:00 +CPU= 16.336 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1998050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0258 (argv4) batchid: 967 (argv5) wuid: 12183902 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/0 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_9f1538260466a9193194173f9ed56c34 to: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip Unzipping the namelist zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip ic_ancil_file: ic_ancil_12183902 ifsdata_file: ifsdata_12183902 climate_data_file: clim_data_12183902 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b3b72de1fea5937ea6fedf6e21b4e9ba to: /var/lib/boinc/slots/0/ic_ancil_12183902.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/0/ic_ancil_12183902.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4f0e1258061265fd68fcb5e19d1d2808 to: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip Unzipping the climate data zip file: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip Checking for progress XML file: /var/lib/boinc/slots/0/progress_file_12183902.xml Opened progress file ok : /var/lib/boinc/slots/0/progress_file_12183902.xml -- Model is restarting -- Adjusting last_iter, 1891, to previous model restart step. Creating progress file: /var/lib/boinc/slots/0/progress_file_12183902.xml last_cpu_time: 23670 upload_file_number: 78 last_iter: 1871 last_upload: 6739200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715 The child process has been launched with process id: 3424569 Executing the command: /var/lib/boinc/slots/0/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 23:54:53 STEP 1872 H=1872:00 +CPU= 19.126 23:55:43 STEP 1873 H=1873:00 +CPU= 10.080 23:55:54 STEP 1874 H=1874:00 +CPU= 9.945 23:56:14 STEP 1875 H=1875:00 +CPU= 14.941 23:56:25 STEP 1876 H=1876:00 +CPU= 10.499 23:56:36 STEP 1877 H=1877:00 +CPU= 10.355 23:56:53 STEP 1878 H=1878:00 +CPU= 16.954 23:57:23 STEP 1879 H=1879:00 +CPU= 10.235 23:57:33 STEP 1880 H=1880:00 +CPU= 9.803 23:57:48 STEP 1881 H=1881:00 +CPU= 14.928 23:57:58 STEP 1882 H=1882:00 +CPU= 9.830 23:58:08 STEP 1883 H=1883:00 +CPU= 9.507 23:58:33 STEP 1884 H=1884:00 +CPU= 16.655 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001884 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001884 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001884 23:58:43 STEP 1885 H=1885:00 +CPU= 10.283 23:58:54 STEP 1886 H=1886:00 +CPU= 10.796 00:01:21 STEP 1887 H=1887:00 +CPU= 17.994 00:02:06 STEP 1888 H=1888:00 +CPU= 10.639 00:02:18 STEP 1889 H=1889:00 +CPU= 9.785 00:02:42 STEP 1890 H=1890:00 +CPU= 15.784 00:02:52 STEP 1891 H=1891:00 +CPU= 9.386 00:03:01 STEP 1892 H=1892:00 +CPU= 9.644 00:03:16 STEP 1893 H=1893:00 +CPU= 14.745 00:03:26 STEP 1894 H=1894:00 +CPU= 9.662 00:03:37 STEP 1895 H=1895:00 +CPU= 9.756 00:04:15 STEP 1896 H=1896:00 +CPU= 19.673 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001896 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001896 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001896 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001872 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001872 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001872 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001884 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001884 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001884 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_78.zip Uploading the intermediate file: upload_file_78.zip 00:05:22 STEP 1897 H=1897:00 +CPU= 10.781 00:05:37 STEP 1898 H=1898:00 +CPU= 11.018 00:05:54 STEP 1899 H=1899:00 +CPU= 17.151 00:06:13 STEP 1900 H=1900:00 +CPU= 10.188 00:06:23 STEP 1901 H=1901:00 +CPU= 9.503 00:06:40 STEP 1902 H=1902:00 +CPU= 15.535 00:06:51 STEP 1903 H=1903:00 +CPU= 10.596 00:07:02 STEP 1904 H=1904:00 +CPU= 10.751 00:07:17 STEP 1905 H=1905:00 +CPU= 15.623 00:08:53 STEP 1906 H=1906:00 +CPU= 10.977 00:09:04 STEP 1907 H=1907:00 +CPU= 10.524 00:09:27 STEP 1908 H=1908:00 +CPU= 18.047 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001908 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001908 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001908 00:09:38 STEP 1909 H=1909:00 +CPU= 10.241 00:11:46 STEP 1910 H=1910:00 +CPU= 10.975 00:12:06 STEP 1911 H=1911:00 +CPU= 16.910 00:12:34 STEP 1912 H=1912:00 +CPU= 10.591 00:12:45 STEP 1913 H=1913:00 +CPU= 9.917 00:13:30 STEP 1914 H=1914:00 +CPU= 16.204 00:13:40 STEP 1915 H=1915:00 +CPU= 9.879 00:13:50 STEP 1916 H=1916:00 +CPU= 10.095 00:14:10 STEP 1917 H=1917:00 +CPU= 15.845 00:14:21 STEP 1918 H=1918:00 +CPU= 10.806 00:14:33 STEP 1919 H=1919:00 +CPU= 11.136 00:15:38 STEP 1920 H=1920:00 +CPU= 20.393 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001920 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001920 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001920 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001896 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001896 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001896 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001908 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001908 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001908 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_79.zip Uploading the intermediate file: upload_file_79.zip 00:15:49 STEP 1921 H=1921:00 +CPU= 11.055 00:16:28 STEP 1922 H=1922:00 +CPU= 10.792 00:16:51 STEP 1923 H=1923:00 +CPU= 16.080 00:17:01 STEP 1924 H=1924:00 +CPU= 10.889 00:19:03 STEP 1925 H=1925:00 +CPU= 11.423 00:19:23 STEP 1926 H=1926:00 +CPU= 17.114 00:19:34 STEP 1927 H=1927:00 +CPU= 10.671 00:19:45 STEP 1928 H=1928:00 +CPU= 11.170 00:20:45 STEP 1929 H=1929:00 +CPU= 16.826 00:21:05 STEP 1930 H=1930:00 +CPU= 10.778 00:21:16 STEP 1931 H=1931:00 +CPU= 10.535 00:22:02 STEP 1932 H=1932:00 +CPU= 18.615 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001932 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001932 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001932 00:22:12 STEP 1933 H=1933:00 +CPU= 10.499 00:22:23 STEP 1934 H=1934:00 +CPU= 10.917 00:23:28 STEP 1935 H=1935:00 +CPU= 16.816 00:28:38 STEP 1936 H=1936:00 +CPU= 12.403 00:30:16 STEP 1937 H=1937:00 +CPU= 10.419 00:30:42 STEP 1938 H=1938:00 +CPU= 15.449 00:30:52 STEP 1939 H=1939:00 +CPU= 9.531 00:31:02 STEP 1940 H=1940:00 +CPU= 9.892 00:31:17 STEP 1941 H=1941:00 +CPU= 15.089 00:31:27 STEP 1942 H=1942:00 +CPU= 9.890 00:31:38 STEP 1943 H=1943:00 +CPU= 10.071 00:36:50 STEP 1944 H=1944:00 +CPU= 21.247 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001944 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001944 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001944 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001920 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001920 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001920 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001932 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001932 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001932 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_80.zip Uploading the intermediate file: upload_file_80.zip 00:38:14 STEP 1945 H=1945:00 +CPU= 11.148 00:40:27 STEP 1946 H=1946:00 +CPU= 11.392 00:41:56 STEP 1947 H=1947:00 +CPU= 17.135 00:42:07 STEP 1948 H=1948:00 +CPU= 11.258 00:42:18 STEP 1949 H=1949:00 +CPU= 11.098 00:42:39 STEP 1950 H=1950:00 +CPU= 16.286 00:42:49 STEP 1951 H=1951:00 +CPU= 9.759 00:43:00 STEP 1952 H=1952:00 +CPU= 10.983 00:44:22 STEP 1953 H=1953:00 +CPU= 16.902 00:44:35 STEP 1954 H=1954:00 +CPU= 11.011 00:46:35 STEP 1955 H=1955:00 +CPU= 10.920 00:49:33 STEP 1956 H=1956:00 +CPU= 18.549 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001956 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001956 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001956 00:49:45 STEP 1957 H=1957:00 +CPU= 10.838 00:50:48 STEP 1958 H=1958:00 +CPU= 11.078 00:51:11 STEP 1959 H=1959:00 +CPU= 15.092 00:51:21 STEP 1960 H=1960:00 +CPU= 10.486 00:51:31 STEP 1961 H=1961:00 +CPU= 10.341 00:51:49 STEP 1962 H=1962:00 +CPU= 17.227 00:52:13 STEP 1963 H=1963:00 +CPU= 10.683 00:52:24 STEP 1964 H=1964:00 +CPU= 9.801 00:52:40 STEP 1965 H=1965:00 +CPU= 15.032 00:52:50 STEP 1966 H=1966:00 +CPU= 9.804 00:53:03 STEP 1967 H=1967:00 +CPU= 9.840 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1998050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0258 (argv4) batchid: 967 (argv5) wuid: 12183902 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/0 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_9f1538260466a9193194173f9ed56c34 to: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip Unzipping the namelist zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip ic_ancil_file: ic_ancil_12183902 ifsdata_file: ifsdata_12183902 climate_data_file: clim_data_12183902 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b3b72de1fea5937ea6fedf6e21b4e9ba to: /var/lib/boinc/slots/0/ic_ancil_12183902.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/0/ic_ancil_12183902.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4f0e1258061265fd68fcb5e19d1d2808 to: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip Unzipping the climate data zip file: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip Checking for progress XML file: /var/lib/boinc/slots/0/progress_file_12183902.xml Opened progress file ok : /var/lib/boinc/slots/0/progress_file_12183902.xml -- Model is restarting -- Adjusting last_iter, 1968, to previous model restart step. Creating progress file: /var/lib/boinc/slots/0/progress_file_12183902.xml last_cpu_time: 24898 upload_file_number: 81 last_iter: 1967 last_upload: 6998400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715 The child process has been launched with process id: 3424745 Executing the command: /var/lib/boinc/slots/0/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 00:58:33 STEP 1968 H=1968:00 +CPU= 19.168 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001968 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001968 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001968 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001944 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001944 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001944 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001956 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001956 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001956 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_81.zip Uploading the intermediate file: upload_file_81.zip 01:00:48 STEP 1969 H=1969:00 +CPU= 11.036 01:01:39 STEP 1970 H=1970:00 +CPU= 11.464 01:01:56 STEP 1971 H=1971:00 +CPU= 16.517 01:03:11 STEP 1972 H=1972:00 +CPU= 11.553 01:03:22 STEP 1973 H=1973:00 +CPU= 10.963 01:03:50 STEP 1974 H=1974:00 +CPU= 16.764 01:05:21 STEP 1975 H=1975:00 +CPU= 10.418 01:05:57 STEP 1976 H=1976:00 +CPU= 10.805 01:08:29 STEP 1977 H=1977:00 +CPU= 22.318 01:08:41 STEP 1978 H=1978:00 +CPU= 11.118 01:08:52 STEP 1979 H=1979:00 +CPU= 11.121 01:11:30 STEP 1980 H=1980:00 +CPU= 18.544 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001980 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001980 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001980 01:12:46 STEP 1981 H=1981:00 +CPU= 10.792 01:12:59 STEP 1982 H=1982:00 +CPU= 11.244 01:13:36 STEP 1983 H=1983:00 +CPU= 16.801 01:14:11 STEP 1984 H=1984:00 +CPU= 11.103 01:14:21 STEP 1985 H=1985:00 +CPU= 10.503 01:14:39 STEP 1986 H=1986:00 +CPU= 17.452 01:15:08 STEP 1987 H=1987:00 +CPU= 10.747 01:15:19 STEP 1988 H=1988:00 +CPU= 11.138 01:15:36 STEP 1989 H=1989:00 +CPU= 17.073 01:15:47 STEP 1990 H=1990:00 +CPU= 10.999 01:15:58 STEP 1991 H=1991:00 +CPU= 11.204 01:16:28 STEP 1992 H=1992:00 +CPU= 19.941 Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001992 Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001992 Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001992 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001968 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001968 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001968 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001980 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001980 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001980 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_82.zip Uploading the intermediate file: upload_file_82.zip 01:16:39 STEP 1993 H=1993:00 +CPU= 10.865 01:16:50 STEP 1994 H=1994:00 +CPU= 11.162 Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:04:14 | 1533721 | 22269101 | oifs_43r3_ps_0258_1998050100_123_967_12183902_0 | 5,616,000 | 19,263 | 0.0034 |
09 May 2023 16:02:42 | 1533721 | 22269101 | oifs_43r3_ps_0258_1998050100_123_967_12183902_0 | 3,974,400 | 13,758 | 0.0035 |
09 May 2023 16:01:52 | 1533721 | 22269101 | oifs_43r3_ps_0258_1998050100_123_967_12183902_0 | 2,678,400 | 9,307 | 0.0035 |
09 May 2023 16:01:48 | 1533721 | 22269101 | oifs_43r3_ps_0258_1998050100_123_967_12183902_0 | 1,728,000 | 5,900 | 0.0034 |
09 May 2023 16:01:44 | 1533721 | 22269101 | oifs_43r3_ps_0258_1998050100_123_967_12183902_0 | 864,000 | 2,879 | 0.0033 |
©2024 cpdn.org