climateprediction.net home page
Task 22294963

Task 22294963

Name oifs_43r3_ps_2247_2021050100_123_947_12165336_2
Workunit 12165336
Created 31 Dec 2022, 22:44:46 UTC
Sent 7 Jan 2023, 0:08:13 UTC
Report deadline 6 Feb 2023, 0:08:13 UTC
Received 11 Jan 2023, 20:08:00 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1534094
Run time 1 days 1 hours 25 min 38 sec
CPU time 23 hours 41 min 41 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 5.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.75 MB
Peak swap size 4,978.16 MB
Peak disk usage 19,151.79 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
20230108:234753:1673250473:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:59923:59923] [20230108:234753:1673250473:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:48:46 STEP 2808 H=2808:00 +CPU= 40.065
  23:49:10 STEP 2809 H=2809:00 +CPU= 22.309
  23:49:35 STEP 2810 H=2810:00 +CPU= 23.241
  23:50:12 STEP 2811 H=2811:00 +CPU= 35.805
  23:50:36 STEP 2812 H=2812:00 +CPU= 22.988
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2247
(argv4) batchid: 947
(argv5) wuid: 12165336
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_108dc147a14c8924245ce7a1cc22ecb6 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2247_2021050100_123_947_12165336.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2247_2021050100_123_947_12165336.zip
ic_ancil_file: ic_ancil_12165336
ifsdata_file: ifsdata_12165336
climate_data_file: clim_data_12165336
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_396de393b616abb14a60370ff61490c8 to: /var/lib/boinc-client/slots/3/ic_ancil_12165336.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12165336.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d214018ddb0ed6803a8d3b04d6ff1210 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165336.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165336.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165336.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165336.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12165336.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12165336.xml
-- Model is restarting --
Adjusting last_iter, 2813, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12165336.xml
last_cpu_time: 81052
upload_file_number: 117
last_iter: 2807
last_upload: 10108800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2247_2021050100_123_947_12165336_2_r1660027845
The child process has been launched with process id: 63063
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:63063:63063] [20230109:133837:1673300317:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:39:50 STEP 2808 H=2808:00 +CPU= 40.469
  13:40:12 STEP 2809 H=2809:00 +CPU= 21.194
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2247
(argv4) batchid: 947
(argv5) wuid: 12165336
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_108dc147a14c8924245ce7a1cc22ecb6 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2247_2021050100_123_947_12165336.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2247_2021050100_123_947_12165336.zip
ic_ancil_file: ic_ancil_12165336
ifsdata_file: ifsdata_12165336
climate_data_file: clim_data_12165336
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_396de393b616abb14a60370ff61490c8 to: /var/lib/boinc-client/slots/3/ic_ancil_12165336.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12165336.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d214018ddb0ed6803a8d3b04d6ff1210 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165336.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165336.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165336.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165336.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12165336.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12165336.xml
-- Model is restarting --
Adjusting last_iter, 2809, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12165336.xml
last_cpu_time: 81110
upload_file_number: 117
last_iter: 2807
last_upload: 10108800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2247_2021050100_123_947_12165336_2_r1660027845
The child process has been launched with process id: 66090
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:66090:66090] [20230110:032820:1673350100:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:29:10 STEP 2808 H=2808:00 +CPU= 41.090
  03:29:33 STEP 2809 H=2809:00 +CPU= 22.647
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2247
(argv4) batchid: 947
(argv5) wuid: 12165336
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_108dc147a14c8924245ce7a1cc22ecb6 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2247_2021050100_123_947_12165336.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2247_2021050100_123_947_12165336.zip
ic_ancil_file: ic_ancil_12165336
ifsdata_file: ifsdata_12165336
climate_data_file: clim_data_12165336
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_396de393b616abb14a60370ff61490c8 to: /var/lib/boinc-client/slots/3/ic_ancil_12165336.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12165336.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d214018ddb0ed6803a8d3b04d6ff1210 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165336.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165336.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165336.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165336.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12165336.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12165336.xml
-- Model is restarting --
Adjusting last_iter, 2810, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12165336.xml
last_cpu_time: 81181
upload_file_number: 117
last_iter: 2807
last_upload: 10108800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2247_2021050100_123_947_12165336_2_r1660027845
The child process has been launched with process id: 1643
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:1643:1643] [20230110:130944:1673384984:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:11:29 STEP 2808 H=2808:00 +CPU= 41.079
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2247
(argv4) batchid: 947
(argv5) wuid: 12165336
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_108dc147a14c8924245ce7a1cc22ecb6 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2247_2021050100_123_947_12165336.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2247_2021050100_123_947_12165336.zip
ic_ancil_file: ic_ancil_12165336
ifsdata_file: ifsdata_12165336
climate_data_file: clim_data_12165336
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_396de393b616abb14a60370ff61490c8 to: /var/lib/boinc-client/slots/3/ic_ancil_12165336.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12165336.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d214018ddb0ed6803a8d3b04d6ff1210 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165336.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165336.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165336.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165336.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12165336.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12165336.xml
-- Model is restarting --
Adjusting last_iter, 2809, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12165336.xml
last_cpu_time: 81236
upload_file_number: 117
last_iter: 2807
last_upload: 10108800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2247_2021050100_123_947_12165336_2_r1660027845
The child process has been launched with process id: 3097
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3097:3097] [20230110:145205:1673391125:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:53:07 STEP 2808 H=2808:00 +CPU= 41.609
  14:53:30 STEP 2809 H=2809:00 +CPU= 22.117
  14:53:54 STEP 2810 H=2810:00 +CPU= 22.866
  14:54:33 STEP 2811 H=2811:00 +CPU= 35.771
  14:54:58 STEP 2812 H=2812:00 +CPU= 23.394
  14:55:21 STEP 2813 H=2813:00 +CPU= 22.121
  14:56:01 STEP 2814 H=2814:00 +CPU= 37.188
  14:56:25 STEP 2815 H=2815:00 +CPU= 22.912
  14:56:51 STEP 2816 H=2816:00 +CPU= 23.520
  14:57:28 STEP 2817 H=2817:00 +CPU= 35.318
  14:57:53 STEP 2818 H=2818:00 +CPU= 23.482
  14:58:17 STEP 2819 H=2819:00 +CPU= 23.027
  14:59:01 STEP 2820 H=2820:00 +CPU= 40.135
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002820
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002820
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002820
  14:59:25 STEP 2821 H=2821:00 +CPU= 22.300
  14:59:49 STEP 2822 H=2822:00 +CPU= 22.900
  15:00:28 STEP 2823 H=2823:00 +CPU= 36.317
  15:00:53 STEP 2824 H=2824:00 +CPU= 22.986
  15:01:15 STEP 2825 H=2825:00 +CPU= 21.393
  15:01:54 STEP 2826 H=2826:00 +CPU= 36.810
  15:02:19 STEP 2827 H=2827:00 +CPU= 22.883
  15:02:43 STEP 2828 H=2828:00 +CPU= 22.770
  15:03:21 STEP 2829 H=2829:00 +CPU= 35.933
  15:03:46 STEP 2830 H=2830:00 +CPU= 23.436
  15:04:10 STEP 2831 H=2831:00 +CPU= 22.759
  15:04:54 STEP 2832 H=2832:00 +CPU= 41.211
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002832
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002832
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2247_2021050100_123_947_12165336_2_r1660027845_117.zip
Uploading the intermediate file: upload_file_117.zip
  15:05:18 STEP 2833 H=2833:00 +CPU= 22.557
  15:05:44 STEP 2834 H=2834:00 +CPU= 23.571
  15:06:21 STEP 2835 H=2835:00 +CPU= 35.279
  15:06:46 STEP 2836 H=2836:00 +CPU= 23.447
  15:07:10 STEP 2837 H=2837:00 +CPU= 22.918
  15:07:48 STEP 2838 H=2838:00 +CPU= 35.830
  15:08:12 STEP 2839 H=2839:00 +CPU= 22.360
  15:08:37 STEP 2840 H=2840:00 +CPU= 23.413
  15:09:17 STEP 2841 H=2841:00 +CPU= 36.829
  15:09:42 STEP 2842 H=2842:00 +CPU= 23.582
  15:10:06 STEP 2843 H=2843:00 +CPU= 22.276
  15:10:47 STEP 2844 H=2844:00 +CPU= 39.017
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002844
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002844
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002844
  15:11:11 STEP 2845 H=2845:00 +CPU= 22.760
  15:11:37 STEP 2846 H=2846:00 +CPU= 23.737
  15:12:15 STEP 2847 H=2847:00 +CPU= 35.746
  15:12:39 STEP 2848 H=2848:00 +CPU= 22.533
  15:13:03 STEP 2849 H=2849:00 +CPU= 22.830
  15:13:43 STEP 2850 H=2850:00 +CPU= 37.461
  15:14:08 STEP 2851 H=2851:00 +CPU= 22.888
  15:14:33 STEP 2852 H=2852:00 +CPU= 23.452
  15:15:11 STEP 2853 H=2853:00 +CPU= 36.619
  15:15:35 STEP 2854 H=2854:00 +CPU= 22.755
  15:15:59 STEP 2855 H=2855:00 +CPU= 22.194
  15:16:44 STEP 2856 H=2856:00 +CPU= 42.163
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002856
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002856
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2247_2021050100_123_947_12165336_2_r1660027845_118.zip
Uploading the intermediate file: upload_file_118.zip
  15:17:09 STEP 2857 H=2857:00 +CPU= 22.776
  15:17:33 STEP 2858 H=2858:00 +CPU= 22.465
  15:18:12 STEP 2859 H=2859:00 +CPU= 36.334
  15:18:36 STEP 2860 H=2860:00 +CPU= 23.156
  15:18:59 STEP 2861 H=2861:00 +CPU= 22.271
  15:19:39 STEP 2862 H=2862:00 +CPU= 37.417
  15:20:02 STEP 2863 H=2863:00 +CPU= 22.273
  15:20:25 STEP 2864 H=2864:00 +CPU= 22.093
  15:21:03 STEP 2865 H=2865:00 +CPU= 35.987
  15:21:29 STEP 2866 H=2866:00 +CPU= 23.464
  15:21:53 STEP 2867 H=2867:00 +CPU= 22.843
  15:22:37 STEP 2868 H=2868:00 +CPU= 40.278
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002868
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002868
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002868
  15:23:01 STEP 2869 H=2869:00 +CPU= 22.962
  15:23:27 STEP 2870 H=2870:00 +CPU= 23.562
  15:24:05 STEP 2871 H=2871:00 +CPU= 35.799
  15:24:28 STEP 2872 H=2872:00 +CPU= 22.164
  15:24:52 STEP 2873 H=2873:00 +CPU= 22.891
  15:25:33 STEP 2874 H=2874:00 +CPU= 37.420
  15:25:57 STEP 2875 H=2875:00 +CPU= 22.762
  15:26:21 STEP 2876 H=2876:00 +CPU= 22.684
  15:27:00 STEP 2877 H=2877:00 +CPU= 36.113
  15:27:25 STEP 2878 H=2878:00 +CPU= 23.419
  15:27:50 STEP 2879 H=2879:00 +CPU= 23.158
  15:28:34 STEP 2880 H=2880:00 +CPU= 41.035
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002880
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002880
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2247_2021050100_123_947_12165336_2_r1660027845_119.zip
Uploading the intermediate file: upload_file_119.zip
  15:28:59 STEP 2881 H=2881:00 +CPU= 22.654
  15:29:24 STEP 2882 H=2882:00 +CPU= 23.672
  15:30:01 STEP 2883 H=2883:00 +CPU= 35.094
  15:30:26 STEP 2884 H=2884:00 +CPU= 22.910
  15:30:50 STEP 2885 H=2885:00 +CPU= 22.644
  15:31:31 STEP 2886 H=2886:00 +CPU= 37.357
  15:31:54 STEP 2887 H=2887:00 +CPU= 22.124
  15:32:17 STEP 2888 H=2888:00 +CPU= 22.151
  15:32:56 STEP 2889 H=2889:00 +CPU= 36.398
  15:33:21 STEP 2890 H=2890:00 +CPU= 23.588
  15:33:45 STEP 2891 H=2891:00 +CPU= 22.863
  15:34:27 STEP 2892 H=2892:00 +CPU= 39.903
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002892
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002892
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002892
  15:34:50 STEP 2893 H=2893:00 +CPU= 22.401
  15:35:15 STEP 2894 H=2894:00 +CPU= 23.082
  15:35:53 STEP 2895 H=2895:00 +CPU= 35.777
  15:36:18 STEP 2896 H=2896:00 +CPU= 23.627
  15:36:43 STEP 2897 H=2897:00 +CPU= 22.823
  15:37:22 STEP 2898 H=2898:00 +CPU= 36.952
  15:37:45 STEP 2899 H=2899:00 +CPU= 21.711
  15:38:10 STEP 2900 H=2900:00 +CPU= 23.576
  15:38:50 STEP 2901 H=2901:00 +CPU= 36.491
  15:39:15 STEP 2902 H=2902:00 +CPU= 23.399
  15:39:41 STEP 2903 H=2903:00 +CPU= 23.882
  15:40:26 STEP 2904 H=2904:00 +CPU= 42.441
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002904
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002904
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2247_2021050100_123_947_12165336_2_r1660027845_120.zip
Uploading the intermediate file: upload_file_120.zip
  15:40:50 STEP 2905 H=2905:00 +CPU= 22.286
  15:41:14 STEP 2906 H=2906:00 +CPU= 22.671
  15:41:54 STEP 2907 H=2907:00 +CPU= 36.571
  15:42:19 STEP 2908 H=2908:00 +CPU= 23.570
  15:42:44 STEP 2909 H=2909:00 +CPU= 23.007
  15:43:23 STEP 2910 H=2910:00 +CPU= 36.605
  15:43:46 STEP 2911 H=2911:00 +CPU= 22.094
  15:44:11 STEP 2912 H=2912:00 +CPU= 23.422
  15:44:51 STEP 2913 H=2913:00 +CPU= 36.483
  15:45:16 STEP 2914 H=2914:00 +CPU= 23.540
  15:45:39 STEP 2915 H=2915:00 +CPU= 22.194
  15:46:22 STEP 2916 H=2916:00 +CPU= 39.578
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002916
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002916
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002916
  15:46:47 STEP 2917 H=2917:00 +CPU= 22.828
  15:47:12 STEP 2918 H=2918:00 +CPU= 23.274
  15:47:49 STEP 2919 H=2919:00 +CPU= 35.512
  15:48:15 STEP 2920 H=2920:00 +CPU= 23.454
  15:48:40 STEP 2921 H=2921:00 +CPU= 23.180
  15:49:20 STEP 2922 H=2922:00 +CPU= 37.483
  15:49:44 STEP 2923 H=2923:00 +CPU= 22.602
  15:50:08 STEP 2924 H=2924:00 +CPU= 22.342
  15:50:47 STEP 2925 H=2925:00 +CPU= 36.510
  15:51:11 STEP 2926 H=2926:00 +CPU= 23.979
  15:51:35 STEP 2927 H=2927:00 +CPU= 22.937
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2247
(argv4) batchid: 947
(argv5) wuid: 12165336
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_108dc147a14c8924245ce7a1cc22ecb6 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2247_2021050100_123_947_12165336.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_2247_2021050100_123_947_12165336.zip
ic_ancil_file: ic_ancil_12165336
ifsdata_file: ifsdata_12165336
climate_data_file: clim_data_12165336
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_396de393b616abb14a60370ff61490c8 to: /var/lib/boinc-client/slots/3/ic_ancil_12165336.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12165336.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d214018ddb0ed6803a8d3b04d6ff1210 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165336.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12165336.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165336.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12165336.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12165336.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12165336.xml
-- Model is restarting --
Adjusting last_iter, 2928, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12165336.xml
last_cpu_time: 84593
upload_file_number: 121
last_iter: 2927
last_upload: 10454400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2247_2021050100_123_947_12165336_2_r1660027845
The child process has been launched with process id: 3545
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-02:1:1:3545:3545] [20230110:155309:1673394789:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:53:56 STEP 2928 H=2928:00 +CPU= 40.641
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002928
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002928
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2247_2021050100_123_947_12165336_2_r1660027845_121.zip
Uploading the intermediate file: upload_file_121.zip
  15:54:20 STEP 2929 H=2929:00 +CPU= 22.774
  15:54:45 STEP 2930 H=2930:00 +CPU= 22.825
  15:55:22 STEP 2931 H=2931:00 +CPU= 35.659
  15:55:47 STEP 2932 H=2932:00 +CPU= 23.082
  15:56:11 STEP 2933 H=2933:00 +CPU= 22.734
  15:56:52 STEP 2934 H=2934:00 +CPU= 37.389
  15:57:15 STEP 2935 H=2935:00 +CPU= 21.977
  15:57:38 STEP 2936 H=2936:00 +CPU= 22.531
  15:58:18 STEP 2937 H=2937:00 +CPU= 36.433
  15:58:43 STEP 2938 H=2938:00 +CPU= 23.609
  15:59:06 STEP 2939 H=2939:00 +CPU= 22.115
  15:59:49 STEP 2940 H=2940:00 +CPU= 39.677
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002940
  16:00:14 STEP 2941 H=2941:00 +CPU= 23.057
  16:00:39 STEP 2942 H=2942:00 +CPU= 23.863
  16:01:16 STEP 2943 H=2943:00 +CPU= 35.430
  16:01:39 STEP 2944 H=2944:00 +CPU= 22.308
  16:02:04 STEP 2945 H=2945:00 +CPU= 22.896
  16:02:44 STEP 2946 H=2946:00 +CPU= 37.491
  16:03:08 STEP 2947 H=2947:00 +CPU= 22.800
  16:03:32 STEP 2948 H=2948:00 +CPU= 22.613
  16:04:11 STEP 2949 H=2949:00 +CPU= 36.021
  16:04:36 STEP 2950 H=2950:00 +CPU= 23.504
  16:05:01 STEP 2951 H=2951:00 +CPU= 23.537
  16:05:46 STEP 2952 H=2952:00 +CPU= 41.194
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhpi1+002952
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhpi1+002952
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhpi1+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMUAhpi1+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMGGhpi1+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165336/ICMSHhpi1+002940
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2247_2021050100_123_947_12165336_2_r1660027845_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
16:09:09 (3542): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:15:07 1534094 22294963 oifs_43r3_ps_2247_2021050100_123_947_12165336_2 10,623,600 85,301 0.0080
09 May 2023 16:11:36 1534094 22294963 oifs_43r3_ps_2247_2021050100_123_947_12165336_2 1,814,400 14,475 0.0080
09 May 2023 16:11:32 1534094 22294963 oifs_43r3_ps_2247_2021050100_123_947_12165336_2 950,400 8,095 0.0085


©2024 cpdn.org