climateprediction.net home page
Task 22295280

Task 22295280

Name oifs_43r3_ps_0674_2019050100_123_988_12205318_1
Workunit 12205318
Created 3 Jan 2023, 2:13:59 UTC
Sent 3 Jan 2023, 3:39:55 UTC
Report deadline 2 Feb 2023, 3:39:55 UTC
Received 4 Jan 2023, 19:27:39 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 148 (0x00000094) Unknown error code
Computer ID 1513103
Run time 17 hours 38 min 52 sec
CPU time 17 hours 22 min 44 sec
Validate state Invalid
Credit 7,257.60
Device peak FLOPS 7.83 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.76 MB
Peak swap size 4,978.08 MB
Peak disk usage 6,356.82 MB
Stderr 
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 148 (0x94, -108)</message>
<stderr_txt>
6.191
  05:22:56 STEP 1636 H=1636:00 +CPU= 22.432
  05:23:15 STEP 1637 H=1637:00 +CPU= 19.626
  05:23:52 STEP 1638 H=1638:00 +CPU= 36.337
  05:24:13 STEP 1639 H=1639:00 +CPU= 21.888
  05:24:36 STEP 1640 H=1640:00 +CPU= 22.165
  05:25:10 STEP 1641 H=1641:00 +CPU= 33.964
  05:25:34 STEP 1642 H=1642:00 +CPU= 24.613
  05:25:56 STEP 1643 H=1643:00 +CPU= 21.849
  05:26:35 STEP 1644 H=1644:00 +CPU= 38.578
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001644
  05:26:57 STEP 1645 H=1645:00 +CPU= 22.012
  05:27:19 STEP 1646 H=1646:00 +CPU= 22.151
  05:27:53 STEP 1647 H=1647:00 +CPU= 34.400
  05:28:15 STEP 1648 H=1648:00 +CPU= 21.436
  05:28:35 STEP 1649 H=1649:00 +CPU= 20.756
  05:29:09 STEP 1650 H=1650:00 +CPU= 33.974
  05:29:33 STEP 1651 H=1651:00 +CPU= 23.113
  05:29:54 STEP 1652 H=1652:00 +CPU= 21.717
  05:30:29 STEP 1653 H=1653:00 +CPU= 35.160
  05:30:51 STEP 1654 H=1654:00 +CPU= 21.598
  05:31:15 STEP 1655 H=1655:00 +CPU= 23.751
  05:31:58 STEP 1656 H=1656:00 +CPU= 43.076
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_68.zip
Uploading the intermediate file: upload_file_68.zip
  05:32:21 STEP 1657 H=1657:00 +CPU= 23.186
  05:32:45 STEP 1658 H=1658:00 +CPU= 23.426
  05:33:19 STEP 1659 H=1659:00 +CPU= 34.812
  05:33:43 STEP 1660 H=1660:00 +CPU= 23.857
  05:34:06 STEP 1661 H=1661:00 +CPU= 22.876
  05:34:40 STEP 1662 H=1662:00 +CPU= 34.248
  05:35:01 STEP 1663 H=1663:00 +CPU= 20.865
  05:35:23 STEP 1664 H=1664:00 +CPU= 21.268
  05:36:00 STEP 1665 H=1665:00 +CPU= 37.238
  05:36:21 STEP 1666 H=1666:00 +CPU= 21.093
  05:36:43 STEP 1667 H=1667:00 +CPU= 22.413
  05:37:24 STEP 1668 H=1668:00 +CPU= 40.894
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001668
  05:37:44 STEP 1669 H=1669:00 +CPU= 19.984
  05:38:06 STEP 1670 H=1670:00 +CPU= 21.412
  05:38:40 STEP 1671 H=1671:00 +CPU= 34.410
  05:39:03 STEP 1672 H=1672:00 +CPU= 23.044
  05:39:23 STEP 1673 H=1673:00 +CPU= 20.205
  05:40:00 STEP 1674 H=1674:00 +CPU= 36.838
  05:40:25 STEP 1675 H=1675:00 +CPU= 24.291
  05:40:47 STEP 1676 H=1676:00 +CPU= 22.318
  05:41:21 STEP 1677 H=1677:00 +CPU= 34.474
  05:41:43 STEP 1678 H=1678:00 +CPU= 22.107
  05:42:08 STEP 1679 H=1679:00 +CPU= 25.001
  05:42:49 STEP 1680 H=1680:00 +CPU= 40.262
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_69.zip
Uploading the intermediate file: upload_file_69.zip
  05:43:12 STEP 1681 H=1681:00 +CPU= 23.254
  05:43:36 STEP 1682 H=1682:00 +CPU= 23.859
  05:44:11 STEP 1683 H=1683:00 +CPU= 34.944
  05:44:32 STEP 1684 H=1684:00 +CPU= 21.576
  05:44:55 STEP 1685 H=1685:00 +CPU= 22.950
  05:45:30 STEP 1686 H=1686:00 +CPU= 34.092
  05:45:52 STEP 1687 H=1687:00 +CPU= 22.436
  05:46:16 STEP 1688 H=1688:00 +CPU= 24.395
  05:46:53 STEP 1689 H=1689:00 +CPU= 36.503
  05:47:14 STEP 1690 H=1690:00 +CPU= 20.792
  05:47:37 STEP 1691 H=1691:00 +CPU= 23.597
  05:48:19 STEP 1692 H=1692:00 +CPU= 41.367
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001692
  05:48:42 STEP 1693 H=1693:00 +CPU= 23.610
  05:49:07 STEP 1694 H=1694:00 +CPU= 25.101
  05:49:46 STEP 1695 H=1695:00 +CPU= 38.365
  05:50:12 STEP 1696 H=1696:00 +CPU= 26.502
  05:50:37 STEP 1697 H=1697:00 +CPU= 24.898
  05:51:17 STEP 1698 H=1698:00 +CPU= 39.191
  05:51:41 STEP 1699 H=1699:00 +CPU= 24.554
  05:52:05 STEP 1700 H=1700:00 +CPU= 24.163
  05:52:42 STEP 1701 H=1701:00 +CPU= 36.597
  05:53:05 STEP 1702 H=1702:00 +CPU= 23.419
  05:53:31 STEP 1703 H=1703:00 +CPU= 25.631
  05:54:20 STEP 1704 H=1704:00 +CPU= 48.769
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_70.zip
Uploading the intermediate file: upload_file_70.zip
  05:54:47 STEP 1705 H=1705:00 +CPU= 26.552
  05:55:13 STEP 1706 H=1706:00 +CPU= 26.217
  05:55:51 STEP 1707 H=1707:00 +CPU= 38.223
  05:56:16 STEP 1708 H=1708:00 +CPU= 24.857
  05:56:42 STEP 1709 H=1709:00 +CPU= 25.598
  05:57:23 STEP 1710 H=1710:00 +CPU= 41.184
  05:57:49 STEP 1711 H=1711:00 +CPU= 25.909
  05:58:15 STEP 1712 H=1712:00 +CPU= 25.931
  05:58:56 STEP 1713 H=1713:00 +CPU= 41.365
  05:59:21 STEP 1714 H=1714:00 +CPU= 24.170
  05:59:47 STEP 1715 H=1715:00 +CPU= 26.537
  06:00:33 STEP 1716 H=1716:00 +CPU= 45.212
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001716
  06:00:57 STEP 1717 H=1717:00 +CPU= 24.590
  06:01:23 STEP 1718 H=1718:00 +CPU= 25.548
  06:02:02 STEP 1719 H=1719:00 +CPU= 39.296
  06:02:27 STEP 1720 H=1720:00 +CPU= 25.384
  06:02:50 STEP 1721 H=1721:00 +CPU= 22.329
  06:03:27 STEP 1722 H=1722:00 +CPU= 37.353
  06:03:51 STEP 1723 H=1723:00 +CPU= 24.337
  06:04:16 STEP 1724 H=1724:00 +CPU= 24.469
  06:04:52 STEP 1725 H=1725:00 +CPU= 35.757
  06:05:13 STEP 1726 H=1726:00 +CPU= 21.314
  06:05:37 STEP 1727 H=1727:00 +CPU= 23.922
  06:06:19 STEP 1728 H=1728:00 +CPU= 42.392
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_71.zip
Uploading the intermediate file: upload_file_71.zip
  06:06:53 STEP 1729 H=1729:00 +CPU= 25.927
  06:07:18 STEP 1730 H=1730:00 +CPU= 24.827
  06:08:01 STEP 1731 H=1731:00 +CPU= 42.546
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0674
(argv4) batchid: 988
(argv5) wuid: 12205318
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c879c1b048f5b0fdf33f76e425997ef5 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0674_2019050100_123_988_12205318.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0674_2019050100_123_988_12205318.zip
ic_ancil_file: ic_ancil_12205318
ifsdata_file: ifsdata_12205318
climate_data_file: clim_data_12205318
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f3bc7aee6cac445b5813614dc114f114 to: /var/lib/boinc-client/slots/11/ic_ancil_12205318.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12205318.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2b8868d07f18625c6eca7a44a928e257 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12205318.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12205318.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12205318.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12205318.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12205318.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12205318.xml
-- Model is restarting --
Adjusting last_iter, 1732, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12205318.xml
last_cpu_time: 55092
upload_file_number: 72
last_iter: 1727
last_upload: 6220800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657
The child process has been launched with process id: 40523
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40523:40523] [20230104:060930:1672808970:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:10:39 STEP 1728 H=1728:00 +CPU= 46.961
  06:11:03 STEP 1729 H=1729:00 +CPU= 24.126
  06:11:26 STEP 1730 H=1730:00 +CPU= 22.339
  06:12:03 STEP 1731 H=1731:00 +CPU= 37.539
  06:12:27 STEP 1732 H=1732:00 +CPU= 23.402
  06:12:50 STEP 1733 H=1733:00 +CPU= 23.261
  06:13:32 STEP 1734 H=1734:00 +CPU= 41.902
  06:13:58 STEP 1735 H=1735:00 +CPU= 26.133
  06:14:23 STEP 1736 H=1736:00 +CPU= 24.512
  06:14:58 STEP 1737 H=1737:00 +CPU= 34.716
  06:15:24 STEP 1738 H=1738:00 +CPU= 26.185
  06:15:51 STEP 1739 H=1739:00 +CPU= 23.513
  06:16:37 STEP 1740 H=1740:00 +CPU= 44.738
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001740
  06:17:00 STEP 1741 H=1741:00 +CPU= 22.220
  06:17:23 STEP 1742 H=1742:00 +CPU= 23.110
  06:18:01 STEP 1743 H=1743:00 +CPU= 37.604
  06:18:27 STEP 1744 H=1744:00 +CPU= 26.246
  06:18:54 STEP 1745 H=1745:00 +CPU= 25.372
  06:19:33 STEP 1746 H=1746:00 +CPU= 39.822
  06:19:56 STEP 1747 H=1747:00 +CPU= 22.231
  06:20:18 STEP 1748 H=1748:00 +CPU= 22.186
  06:20:58 STEP 1749 H=1749:00 +CPU= 39.806
  06:21:24 STEP 1750 H=1750:00 +CPU= 26.074
  06:21:48 STEP 1751 H=1751:00 +CPU= 24.469
  06:22:33 STEP 1752 H=1752:00 +CPU= 43.631
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_72.zip
Uploading the intermediate file: upload_file_72.zip
  06:22:56 STEP 1753 H=1753:00 +CPU= 23.261
  06:23:19 STEP 1754 H=1754:00 +CPU= 23.534
  06:23:57 STEP 1755 H=1755:00 +CPU= 36.293
  06:24:23 STEP 1756 H=1756:00 +CPU= 25.850
  06:24:49 STEP 1757 H=1757:00 +CPU= 25.642
  06:25:30 STEP 1758 H=1758:00 +CPU= 41.564
  06:25:57 STEP 1759 H=1759:00 +CPU= 22.959
  06:26:19 STEP 1760 H=1760:00 +CPU= 22.482
  06:26:56 STEP 1761 H=1761:00 +CPU= 37.060
  06:27:20 STEP 1762 H=1762:00 +CPU= 23.828
  06:27:44 STEP 1763 H=1763:00 +CPU= 23.967
  06:28:24 STEP 1764 H=1764:00 +CPU= 39.504
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001764
  06:28:48 STEP 1765 H=1765:00 +CPU= 23.737
  06:29:12 STEP 1766 H=1766:00 +CPU= 24.717
  06:29:54 STEP 1767 H=1767:00 +CPU= 39.759
  06:30:20 STEP 1768 H=1768:00 +CPU= 25.985
  06:30:46 STEP 1769 H=1769:00 +CPU= 25.430
  06:31:22 STEP 1770 H=1770:00 +CPU= 36.390
  06:31:46 STEP 1771 H=1771:00 +CPU= 23.508
  06:32:08 STEP 1772 H=1772:00 +CPU= 22.565
  06:32:45 STEP 1773 H=1773:00 +CPU= 34.548
  06:33:11 STEP 1774 H=1774:00 +CPU= 24.206
  06:33:36 STEP 1775 H=1775:00 +CPU= 25.427
  06:34:36 STEP 1776 H=1776:00 +CPU= 43.441
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_73.zip
Uploading the intermediate file: upload_file_73.zip
  06:35:00 STEP 1777 H=1777:00 +CPU= 24.746
  06:35:23 STEP 1778 H=1778:00 +CPU= 21.247
  06:36:05 STEP 1779 H=1779:00 +CPU= 36.159
  06:36:29 STEP 1780 H=1780:00 +CPU= 23.861
  06:36:51 STEP 1781 H=1781:00 +CPU= 22.159
  06:37:36 STEP 1782 H=1782:00 +CPU= 36.457
  06:37:57 STEP 1783 H=1783:00 +CPU= 21.631
  06:38:26 STEP 1784 H=1784:00 +CPU= 25.729
  06:39:02 STEP 1785 H=1785:00 +CPU= 36.287
  06:39:26 STEP 1786 H=1786:00 +CPU= 24.153
  06:39:55 STEP 1787 H=1787:00 +CPU= 22.966
  06:40:38 STEP 1788 H=1788:00 +CPU= 41.540
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001788
  06:40:59 STEP 1789 H=1789:00 +CPU= 21.008
  06:41:28 STEP 1790 H=1790:00 +CPU= 25.481
  06:42:07 STEP 1791 H=1791:00 +CPU= 38.285
  06:42:32 STEP 1792 H=1792:00 +CPU= 24.507
  06:42:54 STEP 1793 H=1793:00 +CPU= 22.788
  06:43:29 STEP 1794 H=1794:00 +CPU= 34.953
  06:43:55 STEP 1795 H=1795:00 +CPU= 25.773
  06:44:19 STEP 1796 H=1796:00 +CPU= 23.424
  06:44:55 STEP 1797 H=1797:00 +CPU= 36.246
  06:45:19 STEP 1798 H=1798:00 +CPU= 24.311
  06:45:45 STEP 1799 H=1799:00 +CPU= 24.065
  06:46:30 STEP 1800 H=1800:00 +CPU= 40.107
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_74.zip
Uploading the intermediate file: upload_file_74.zip
  06:46:52 STEP 1801 H=1801:00 +CPU= 22.799
  06:47:17 STEP 1802 H=1802:00 +CPU= 24.229
  06:47:58 STEP 1803 H=1803:00 +CPU= 39.559
  06:48:21 STEP 1804 H=1804:00 +CPU= 22.912
  06:48:42 STEP 1805 H=1805:00 +CPU= 21.049
  06:49:38 STEP 1806 H=1806:00 +CPU= 36.383
  06:50:02 STEP 1807 H=1807:00 +CPU= 22.798
  06:50:26 STEP 1808 H=1808:00 +CPU= 24.324
  06:51:04 STEP 1809 H=1809:00 +CPU= 37.060
  06:51:29 STEP 1810 H=1810:00 +CPU= 24.721
  06:51:53 STEP 1811 H=1811:00 +CPU= 24.365
  06:52:42 STEP 1812 H=1812:00 +CPU= 40.658
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001812
  06:53:07 STEP 1813 H=1813:00 +CPU= 24.488
  06:53:34 STEP 1814 H=1814:00 +CPU= 24.111
  06:54:10 STEP 1815 H=1815:00 +CPU= 35.598
  06:54:34 STEP 1816 H=1816:00 +CPU= 24.288
  06:54:58 STEP 1817 H=1817:00 +CPU= 24.590
  06:55:37 STEP 1818 H=1818:00 +CPU= 38.126
  06:56:00 STEP 1819 H=1819:00 +CPU= 23.777
  06:56:25 STEP 1820 H=1820:00 +CPU= 24.602
  06:57:02 STEP 1821 H=1821:00 +CPU= 37.203
  06:57:25 STEP 1822 H=1822:00 +CPU= 22.858
  06:57:59 STEP 1823 H=1823:00 +CPU= 27.895
  06:58:50 STEP 1824 H=1824:00 +CPU= 44.400
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_75.zip
Uploading the intermediate file: upload_file_75.zip
  06:59:14 STEP 1825 H=1825:00 +CPU= 23.764
  06:59:39 STEP 1826 H=1826:00 +CPU= 25.317
  07:00:17 STEP 1827 H=1827:00 +CPU= 38.357
  07:00:41 STEP 1828 H=1828:00 +CPU= 24.080
  07:01:05 STEP 1829 H=1829:00 +CPU= 23.958
  07:01:45 STEP 1830 H=1830:00 +CPU= 39.814
  07:02:11 STEP 1831 H=1831:00 +CPU= 25.685
  07:02:36 STEP 1832 H=1832:00 +CPU= 24.892
  07:03:14 STEP 1833 H=1833:00 +CPU= 38.291
  07:03:37 STEP 1834 H=1834:00 +CPU= 22.969
  07:04:00 STEP 1835 H=1835:00 +CPU= 22.444
  07:04:42 STEP 1836 H=1836:00 +CPU= 40.292
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001836
  07:05:06 STEP 1837 H=1837:00 +CPU= 22.899
  07:05:31 STEP 1838 H=1838:00 +CPU= 25.167
  07:06:07 STEP 1839 H=1839:00 +CPU= 35.322
  07:06:31 STEP 1840 H=1840:00 +CPU= 24.294
  07:06:58 STEP 1841 H=1841:00 +CPU= 26.994
  07:07:37 STEP 1842 H=1842:00 +CPU= 39.436
  07:08:01 STEP 1843 H=1843:00 +CPU= 23.769
  07:08:29 STEP 1844 H=1844:00 +CPU= 27.534
  07:09:15 STEP 1845 H=1845:00 +CPU= 40.852
  07:09:41 STEP 1846 H=1846:00 +CPU= 24.910
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0674
(argv4) batchid: 988
(argv5) wuid: 12205318
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c879c1b048f5b0fdf33f76e425997ef5 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0674_2019050100_123_988_12205318.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0674_2019050100_123_988_12205318.zip
ic_ancil_file: ic_ancil_12205318
ifsdata_file: ifsdata_12205318
climate_data_file: clim_data_12205318
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f3bc7aee6cac445b5813614dc114f114 to: /var/lib/boinc-client/slots/11/ic_ancil_12205318.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12205318.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2b8868d07f18625c6eca7a44a928e257 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12205318.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12205318.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12205318.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12205318.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12205318.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12205318.xml
-- Model is restarting --
Adjusting last_iter, 1847, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12205318.xml
last_cpu_time: 58565
upload_file_number: 76
last_iter: 1823
last_upload: 6566400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657
The child process has been launched with process id: 40904
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:40904:40904] [20230104:071208:1672812728:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:13:39 STEP 1824 H=1824:00 +CPU= 42.465
  07:14:03 STEP 1825 H=1825:00 +CPU= 23.338
  07:14:26 STEP 1826 H=1826:00 +CPU= 22.950
  07:15:03 STEP 1827 H=1827:00 +CPU= 37.630
  07:15:28 STEP 1828 H=1828:00 +CPU= 25.055
  07:15:52 STEP 1829 H=1829:00 +CPU= 23.891
  07:16:30 STEP 1830 H=1830:00 +CPU= 37.558
  07:16:59 STEP 1831 H=1831:00 +CPU= 24.297
  07:17:23 STEP 1832 H=1832:00 +CPU= 23.602
  07:18:01 STEP 1833 H=1833:00 +CPU= 37.821
  07:18:36 STEP 1834 H=1834:00 +CPU= 25.171
  07:18:58 STEP 1835 H=1835:00 +CPU= 22.515
  07:19:39 STEP 1836 H=1836:00 +CPU= 39.267
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001836
  07:20:02 STEP 1837 H=1837:00 +CPU= 22.948
  07:20:25 STEP 1838 H=1838:00 +CPU= 23.244
  07:21:01 STEP 1839 H=1839:00 +CPU= 35.921
  07:21:26 STEP 1840 H=1840:00 +CPU= 25.161
  07:21:50 STEP 1841 H=1841:00 +CPU= 24.019
  07:22:25 STEP 1842 H=1842:00 +CPU= 35.365
  07:22:51 STEP 1843 H=1843:00 +CPU= 26.061
  07:23:17 STEP 1844 H=1844:00 +CPU= 25.192
  07:24:06 STEP 1845 H=1845:00 +CPU= 38.871
  07:24:35 STEP 1846 H=1846:00 +CPU= 28.331
  07:25:02 STEP 1847 H=1847:00 +CPU= 25.585
  07:26:11 STEP 1848 H=1848:00 +CPU= 44.756
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_76.zip
Uploading the intermediate file: upload_file_76.zip
  07:26:36 STEP 1849 H=1849:00 +CPU= 22.750
  07:27:03 STEP 1850 H=1850:00 +CPU= 24.620
  07:27:42 STEP 1851 H=1851:00 +CPU= 38.862
  07:28:05 STEP 1852 H=1852:00 +CPU= 22.745
  07:28:31 STEP 1853 H=1853:00 +CPU= 25.225
  07:29:07 STEP 1854 H=1854:00 +CPU= 35.890
  07:29:29 STEP 1855 H=1855:00 +CPU= 21.858
  07:29:54 STEP 1856 H=1856:00 +CPU= 25.615
  07:30:33 STEP 1857 H=1857:00 +CPU= 39.003
  07:30:59 STEP 1858 H=1858:00 +CPU= 25.461
  07:31:24 STEP 1859 H=1859:00 +CPU= 24.819
  07:32:05 STEP 1860 H=1860:00 +CPU= 40.744
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001860
  07:32:27 STEP 1861 H=1861:00 +CPU= 22.174
  07:32:50 STEP 1862 H=1862:00 +CPU= 22.609
  07:33:27 STEP 1863 H=1863:00 +CPU= 37.556
  07:33:51 STEP 1864 H=1864:00 +CPU= 23.728
  07:34:14 STEP 1865 H=1865:00 +CPU= 23.420
  07:34:58 STEP 1866 H=1866:00 +CPU= 39.891
  07:35:21 STEP 1867 H=1867:00 +CPU= 23.203
  07:35:44 STEP 1868 H=1868:00 +CPU= 23.693
  07:36:23 STEP 1869 H=1869:00 +CPU= 38.494
  07:36:50 STEP 1870 H=1870:00 +CPU= 26.532
  07:37:13 STEP 1871 H=1871:00 +CPU= 22.917
  07:37:59 STEP 1872 H=1872:00 +CPU= 46.037
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_77.zip
Uploading the intermediate file: upload_file_77.zip
  07:38:29 STEP 1873 H=1873:00 +CPU= 20.626
  07:38:53 STEP 1874 H=1874:00 +CPU= 24.498
  07:39:30 STEP 1875 H=1875:00 +CPU= 36.849
  07:39:52 STEP 1876 H=1876:00 +CPU= 22.451
  07:40:17 STEP 1877 H=1877:00 +CPU= 24.749
  07:41:01 STEP 1878 H=1878:00 +CPU= 39.418
  07:41:38 STEP 1879 H=1879:00 +CPU= 22.888
  07:42:02 STEP 1880 H=1880:00 +CPU= 23.377
  07:42:38 STEP 1881 H=1881:00 +CPU= 36.349
  07:43:05 STEP 1882 H=1882:00 +CPU= 26.310
  07:43:28 STEP 1883 H=1883:00 +CPU= 23.904
  07:44:14 STEP 1884 H=1884:00 +CPU= 43.193
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001884
  07:44:38 STEP 1885 H=1885:00 +CPU= 23.633
  07:45:00 STEP 1886 H=1886:00 +CPU= 22.377
  07:45:48 STEP 1887 H=1887:00 +CPU= 38.473
  07:46:15 STEP 1888 H=1888:00 +CPU= 26.697
  07:46:39 STEP 1889 H=1889:00 +CPU= 23.940
  07:47:19 STEP 1890 H=1890:00 +CPU= 39.415
  07:47:43 STEP 1891 H=1891:00 +CPU= 24.041
  07:48:08 STEP 1892 H=1892:00 +CPU= 24.398
  07:48:48 STEP 1893 H=1893:00 +CPU= 40.136
  07:49:11 STEP 1894 H=1894:00 +CPU= 23.620
  07:49:39 STEP 1895 H=1895:00 +CPU= 26.855
  07:50:29 STEP 1896 H=1896:00 +CPU= 42.870
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_78.zip
Uploading the intermediate file: upload_file_78.zip
  07:50:52 STEP 1897 H=1897:00 +CPU= 22.888
  07:51:19 STEP 1898 H=1898:00 +CPU= 27.371
  07:51:56 STEP 1899 H=1899:00 +CPU= 36.458
  07:52:18 STEP 1900 H=1900:00 +CPU= 22.674
  07:52:41 STEP 1901 H=1901:00 +CPU= 22.291
  07:53:20 STEP 1902 H=1902:00 +CPU= 38.770
  07:53:41 STEP 1903 H=1903:00 +CPU= 21.314
  07:54:05 STEP 1904 H=1904:00 +CPU= 23.937
  07:54:44 STEP 1905 H=1905:00 +CPU= 38.998
  07:55:10 STEP 1906 H=1906:00 +CPU= 25.675
  07:55:35 STEP 1907 H=1907:00 +CPU= 25.279
  07:56:19 STEP 1908 H=1908:00 +CPU= 42.832
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001908
  07:56:43 STEP 1909 H=1909:00 +CPU= 24.020
  07:57:10 STEP 1910 H=1910:00 +CPU= 26.271
  07:57:48 STEP 1911 H=1911:00 +CPU= 37.536
  07:58:13 STEP 1912 H=1912:00 +CPU= 24.782
  07:58:36 STEP 1913 H=1913:00 +CPU= 23.072
  07:59:08 STEP 1914 H=1914:00 +CPU= 32.000
  07:59:28 STEP 1915 H=1915:00 +CPU= 20.285
  07:59:51 STEP 1916 H=1916:00 +CPU= 23.355
  08:00:30 STEP 1917 H=1917:00 +CPU= 38.458
  08:00:55 STEP 1918 H=1918:00 +CPU= 24.737
  08:01:19 STEP 1919 H=1919:00 +CPU= 23.986
  08:02:01 STEP 1920 H=1920:00 +CPU= 42.164
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657_79.zip
Uploading the intermediate file: upload_file_79.zip
  08:02:25 STEP 1921 H=1921:00 +CPU= 24.109
  08:02:51 STEP 1922 H=1922:00 +CPU= 25.176
  08:03:29 STEP 1923 H=1923:00 +CPU= 38.036
  08:03:53 STEP 1924 H=1924:00 +CPU= 24.507
  08:04:19 STEP 1925 H=1925:00 +CPU= 24.848
  08:04:55 STEP 1926 H=1926:00 +CPU= 35.379
  08:05:15 STEP 1927 H=1927:00 +CPU= 20.320
  08:05:38 STEP 1928 H=1928:00 +CPU= 22.859
  08:06:15 STEP 1929 H=1929:00 +CPU= 36.465
  08:06:41 STEP 1930 H=1930:00 +CPU= 25.033
  08:07:16 STEP 1931 H=1931:00 +CPU= 25.925
  08:08:06 STEP 1932 H=1932:00 +CPU= 42.156
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001932
  08:08:32 STEP 1933 H=1933:00 +CPU= 24.940
  08:09:07 STEP 1934 H=1934:00 +CPU= 23.717
  08:09:45 STEP 1935 H=1935:00 +CPU= 37.894
  08:10:11 STEP 1936 H=1936:00 +CPU= 25.452
  08:10:49 STEP 1937 H=1937:00 +CPU= 23.613
  08:11:29 STEP 1938 H=1938:00 +CPU= 39.529
  08:11:52 STEP 1939 H=1939:00 +CPU= 22.489
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0674
(argv4) batchid: 988
(argv5) wuid: 12205318
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c879c1b048f5b0fdf33f76e425997ef5 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0674_2019050100_123_988_12205318.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0674_2019050100_123_988_12205318.zip
ic_ancil_file: ic_ancil_12205318
ifsdata_file: ifsdata_12205318
climate_data_file: clim_data_12205318
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f3bc7aee6cac445b5813614dc114f114 to: /var/lib/boinc-client/slots/11/ic_ancil_12205318.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12205318.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2b8868d07f18625c6eca7a44a928e257 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12205318.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12205318.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12205318.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12205318.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12205318.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12205318.xml
-- Model is restarting --
Adjusting last_iter, 1940, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12205318.xml
last_cpu_time: 61961
upload_file_number: 80
last_iter: 1919
last_upload: 6912000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0674_2019050100_123_988_12205318_1_r234770657
The child process has been launched with process id: 43779
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:43779:43779] [20230104:081525:1672816525:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:17:19 STEP 1920 H=1920:00 +CPU= 43.222
  08:17:44 STEP 1921 H=1921:00 +CPU= 24.681
  08:18:20 STEP 1922 H=1922:00 +CPU= 25.595
  08:18:56 STEP 1923 H=1923:00 +CPU= 35.691
  08:19:21 STEP 1924 H=1924:00 +CPU= 24.924
  08:19:51 STEP 1925 H=1925:00 +CPU= 24.261
  08:20:29 STEP 1926 H=1926:00 +CPU= 37.577
  08:20:52 STEP 1927 H=1927:00 +CPU= 23.355
  08:21:21 STEP 1928 H=1928:00 +CPU= 23.700
  08:22:12 STEP 1929 H=1929:00 +CPU= 34.563
  08:22:37 STEP 1930 H=1930:00 +CPU= 24.223
  08:23:00 STEP 1931 H=1931:00 +CPU= 23.049
  08:23:53 STEP 1932 H=1932:00 +CPU= 45.158
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001932
  08:24:17 STEP 1933 H=1933:00 +CPU= 23.829
  08:24:41 STEP 1934 H=1934:00 +CPU= 23.995
  08:25:19 STEP 1935 H=1935:00 +CPU= 38.187
  08:25:54 STEP 1936 H=1936:00 +CPU= 25.136
  08:26:19 STEP 1937 H=1937:00 +CPU= 24.285
  08:26:56 STEP 1938 H=1938:00 +CPU= 37.567
  08:27:21 STEP 1939 H=1939:00 +CPU= 24.139
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0674
(argv4) batchid: 988
(argv5) wuid: 12205318
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c879c1b048f5b0fdf33f76e425997ef5 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0674_2019050100_123_988_12205318.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0674_2019050100_123_988_12205318.zip
ic_ancil_file: ic_ancil_12205318
ifsdata_file: ifsdata_12205318
climate_data_file: clim_data_12205318
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f3bc7aee6cac445b5813614dc114f114 to: /var/lib/boinc-client/slots/11/ic_ancil_12205318.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12205318.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0674
(argv4) batchid: 988
(argv5) wuid: 12205318
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205318
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c879c1b048f5b0fdf33f76e425997ef5 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0674_2019050100_123_988_12205318.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0674_2019050100_123_988_12205318.zip
ic_ancil_file: ic_ancil_12205318
ifsdata_file: ifsdata_12205318
climate_data_file: clim_data_12205318
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from:  to: /var/lib/boinc-client/slots/11/ic_ancil_12205318.zip
..Copying the IC ancils to the working directory failed

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:07:07 1513103 22295280 oifs_43r3_ps_0674_2019050100_123_988_12205318_1 5,184,000 44,745 0.0086
09 May 2023 16:06:59 1513103 22295280 oifs_43r3_ps_0674_2019050100_123_988_12205318_1 4,320,000 37,269 0.0086
09 May 2023 16:06:55 1513103 22295280 oifs_43r3_ps_0674_2019050100_123_988_12205318_1 3,456,000 29,856 0.0086
09 May 2023 16:06:51 1513103 22295280 oifs_43r3_ps_0674_2019050100_123_988_12205318_1 2,592,000 22,106 0.0085
09 May 2023 16:06:51 1513103 22295280 oifs_43r3_ps_0674_2019050100_123_988_12205318_1 2,592,000 22,106 0.0085
09 May 2023 16:06:46 1513103 22295280 oifs_43r3_ps_0674_2019050100_123_988_12205318_1 1,728,000 14,937 0.0086
09 May 2023 16:06:42 1513103 22295280 oifs_43r3_ps_0674_2019050100_123_988_12205318_1 864,000 7,789 0.0090
09 May 2023 16:06:42 1513103 22295280 oifs_43r3_ps_0674_2019050100_123_988_12205318_1 864,000 7,789 0.0090


©2024 climateprediction.net