<core_client_version>7.24.1</core_client_version>
<![CDATA[
<message>
process exited with code 148 (0x94, -108)</message>
<stderr_txt>
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0ag
(argv4) batchid: 1017
(argv5) wuid: 12282203
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/1
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12282203
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f5d119923a679558d4e0f70ded02c0ea to: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
Unzipping the namelist zip file: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
ic_ancil_file: ic_ancil_12282203
ifsdata_file: ifsdata_12282203
climate_data_file: clim_data_12282203
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/1/ic_ancil_12282203.zip
Unzipping the IC ancils zip file: /home/i3/slots/1/ic_ancil_12282203.zip
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_303b81978d4a6c56a2ab71309c463448 to: /home/i3/slots/1/ifsdata/ifsdata_12282203.zip
Unzipping the ifsdata_zip file: /home/i3/slots/1/ifsdata/ifsdata_12282203.zip
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/1/159l_2/clim_data_12282203.zip
Unzipping the climate data zip file: /home/i3/slots/1/159l_2/clim_data_12282203.zip
Checking for progress XML file: /home/i3/slots/1/progress_file_12282203.xml
Creating progress file: /home/i3/slots/1/progress_file_12282203.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a0ag_2016092300_20_1017_12282203_0_r983228653
The child process has been launched with process id: 63496
Executing the command: /home/i3/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:63496:63496] [20240607:133631:1717760191:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0ag
(argv4) batchid: 1017
(argv5) wuid: 12282203
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/1
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12282203
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f5d119923a679558d4e0f70ded02c0ea to: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
Unzipping the namelist zip file: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
ic_ancil_file: ic_ancil_12282203
ifsdata_file: ifsdata_12282203
climate_data_file: clim_data_12282203
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/1/ic_ancil_12282203.zip
Unzipping the IC ancils zip file: /home/i3/slots/1/ic_ancil_12282203.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_303b81978d4a6c56a2ab71309c463448 to: /home/i3/slots/1/ifsdata/ifsdata_12282203.zip
Unzipping the ifsdata_zip file: /home/i3/slots/1/ifsdata/ifsdata_12282203.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/1/159l_2/clim_data_12282203.zip
Unzipping the climate data zip file: /home/i3/slots/1/159l_2/clim_data_12282203.zip
Checking for progress XML file: /home/i3/slots/1/progress_file_12282203.xml
Opened progress file ok : /home/i3/slots/1/progress_file_12282203.xml
-- Model is restarting --
Adjusting last_iter, 0, to previous model restart step.
Creating progress file: /home/i3/slots/1/progress_file_12282203.xml
last_cpu_time: 18
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a0ag_2016092300_20_1017_12282203_0_r983228653
The child process has been launched with process id: 9567
Executing the command: /home/i3/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9567:9567] [20240607:201619:1717784179:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:16:57 STEP    0 H=   0:00 +CPU= 23.995
Moving to projects directory: /home/i3/slots/1/ICMGGh7zg+000000
Moving to projects directory: /home/i3/slots/1/ICMSHh7zg+000000
Moving to projects directory: /home/i3/slots/1/ICMUAh7zg+000000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:17:20 STEP    1 H=   0:15 +CPU= 19.164
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:17:45 STEP    2 H=   0:30 +CPU= 18.780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:20:09 STEP    3 H=   0:45 +CPU= 16.233
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:20:29 STEP    4 H=   1:00 +CPU= 16.263
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:20:49 STEP    5 H=   1:15 +CPU= 15.621
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:21:08 STEP    6 H=   1:30 +CPU= 15.350
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:21:29 STEP    7 H=   1:45 +CPU= 16.366
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0ag
(argv4) batchid: 1017
(argv5) wuid: 12282203
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/1
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12282203
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
  20:48:32 STEP    8 H=   2:00 +CPU= 15.993
Unzipping the app zip file: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f5d119923a679558d4e0f70ded02c0ea to: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
Unzipping the namelist zip file: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
ic_ancil_file: ic_ancil_12282203
ifsdata_file: ifsdata_12282203
climate_data_file: clim_data_12282203
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/1/ic_ancil_12282203.zip
Unzipping the IC ancils zip file: /home/i3/slots/1/ic_ancil_12282203.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_303b81978d4a6c56a2ab71309c463448 to: /home/i3/slots/1/ifsdata/ifsdata_12282203.zip
Unzipping the ifsdata_zip file: /home/i3/slots/1/ifsdata/ifsdata_12282203.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/1/159l_2/clim_data_12282203.zip
Unzipping the climate data zip file: /home/i3/slots/1/159l_2/clim_data_12282203.zip
Checking for progress XML file: /home/i3/slots/1/progress_file_12282203.xml
Opened progress file ok : /home/i3/slots/1/progress_file_12282203.xml
-- Model is restarting --
Adjusting last_iter, 8, to previous model restart step.
Creating progress file: /home/i3/slots/1/progress_file_12282203.xml
last_cpu_time: 174
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a0ag_2016092300_20_1017_12282203_0_r983228653
The child process has been launched with process id: 9704
Executing the command: /home/i3/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9704:9704] [20240607:204842:1717786122:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:48:43 STEP    9 H=   2:15 +CPU= 10.305
Suspend request received from the BOINC client, suspending the child process
  20:48:53 STEP   10 H=   2:30 +CPU=  9.643
Resuming the child process
  20:49:06 STEP   11 H=   2:45 +CPU= 13.412
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0ag
(argv4) batchid: 1017
(argv5) wuid: 12282203
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/1
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12282203
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f5d119923a679558d4e0f70ded02c0ea to: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
Unzipping the namelist zip file: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
ic_ancil_file: ic_ancil_12282203
ifsdata_file: ifsdata_12282203
climate_data_file: clim_data_12282203
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/1/ic_ancil_12282203.zip
Unzipping the IC ancils zip file: /home/i3/slots/1/ic_ancil_12282203.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_303b81978d4a6c56a2ab71309c463448 to: /home/i3/slots/1/ifsdata/ifsdata_12282203.zip
Unzipping the ifsdata_zip file: /home/i3/slots/1/ifsdata/ifsdata_12282203.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/1/159l_2/clim_data_12282203.zip
Unzipping the climate data zip file: /home/i3/slots/1/159l_2/clim_data_12282203.zip
Checking for progress XML file: /home/i3/slots/1/progress_file_12282203.xml
Opened progress file ok : /home/i3/slots/1/progress_file_12282203.xml
-- Model is restarting --
Adjusting last_iter, 12, to previous model restart step.
Creating progress file: /home/i3/slots/1/progress_file_12282203.xml
last_cpu_time: 187
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a0ag_2016092300_20_1017_12282203_0_r983228653
The child process has been launched with process id: 9931
Executing the command: /home/i3/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:9931:9931] [20240607:224813:1717793293:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0ag
(argv4) batchid: 1017
(argv5) wuid: 12282203
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/1
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12282203
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f5d119923a679558d4e0f70ded02c0ea to: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
Unzipping the namelist zip file: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
ic_ancil_file: ic_ancil_12282203
ifsdata_file: ifsdata_12282203
climate_data_file: clim_data_12282203
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/1/ic_ancil_12282203.zip
Unzipping the IC ancils zip file: /home/i3/slots/1/ic_ancil_12282203.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_303b81978d4a6c56a2ab71309c463448 to: /home/i3/slots/1/ifsdata/ifsdata_12282203.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0ag
(argv4) batchid: 1017
(argv5) wuid: 12282203
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/1
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12282203
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f5d119923a679558d4e0f70ded02c0ea to: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
Unzipping the namelist zip file: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
ic_ancil_file: ic_ancil_12282203
ifsdata_file: ifsdata_12282203
climate_data_file: clim_data_12282203
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/1/ic_ancil_12282203.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0ag
(argv4) batchid: 1017
(argv5) wuid: 12282203
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/1
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12282203
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f5d119923a679558d4e0f70ded02c0ea to: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
Unzipping the namelist zip file: /home/i3/slots/1/oifs_43r3_bl_a0ag_2016092300_20_1017_12282203.zip
ic_ancil_file: ic_ancil_12282203
ifsdata_file: ifsdata_12282203
climate_data_file: clim_data_12282203
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from:  to: /home/i3/slots/1/ic_ancil_12282203.zip
..Copying the IC ancils to the working directory failed

</stderr_txt>
]]>