climateprediction.net home page
Task 22446313

Task 22446313

Name oifs_43r3_bl_a3dl_2016092300_20_1018_12292704_0
Workunit 12292704
Created 13 Jun 2024, 2:21:39 UTC
Sent 14 Jun 2024, 8:47:49 UTC
Report deadline 13 Aug 2024, 8:47:49 UTC
Received 14 Jun 2024, 18:51:25 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1550987
Run time 11 min 15 sec
CPU time 2 min 45 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,451.52 MB
Peak swap size 5,981.08 MB
Peak disk usage 310.77 MB
Stderr
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:07:03 STEP    0 H=   0:00 +CPU= 22.508
Moving to projects directory: /home/i3/slots/3/ICMGGh7zg+000000
Moving to projects directory: /home/i3/slots/3/ICMSHh7zg+000000
Moving to projects directory: /home/i3/slots/3/ICMUAh7zg+000000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3dl
(argv4) batchid: 1018
(argv5) wuid: 12292704
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/3
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12292704
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b23ff273b95b2d3d49265b2d6d178b19 to: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
Unzipping the namelist zip file: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
ic_ancil_file: ic_ancil_12292704
ifsdata_file: ifsdata_12292704
climate_data_file: clim_data_12292704
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/3/ic_ancil_12292704.zip
Unzipping the IC ancils zip file: /home/i3/slots/3/ic_ancil_12292704.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e1c37b5d8581da24b85a4837aa9564a8 to: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
Unzipping the ifsdata_zip file: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Unzipping the climate data zip file: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Checking for progress XML file: /home/i3/slots/3/progress_file_12292704.xml
Opened progress file ok : /home/i3/slots/3/progress_file_12292704.xml
-- Model is restarting --
Adjusting last_iter, 1, to previous model restart step.
Creating progress file: /home/i3/slots/3/progress_file_12292704.xml
last_cpu_time: 34
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3dl_2016092300_20_1018_12292704_0_r1920888991
The child process has been launched with process id: 10239
Executing the command: /home/i3/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3dl
(argv4) batchid: 1018
(argv5) wuid: 12292704
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/3
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12292704
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b23ff273b95b2d3d49265b2d6d178b19 to: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
Unzipping the namelist zip file: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
ic_ancil_file: ic_ancil_12292704
ifsdata_file: ifsdata_12292704
climate_data_file: clim_data_12292704
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/3/ic_ancil_12292704.zip
Unzipping the IC ancils zip file: /home/i3/slots/3/ic_ancil_12292704.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e1c37b5d8581da24b85a4837aa9564a8 to: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
Unzipping the ifsdata_zip file: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Unzipping the climate data zip file: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Checking for progress XML file: /home/i3/slots/3/progress_file_12292704.xml
Opened progress file ok : /home/i3/slots/3/progress_file_12292704.xml
-- Model is restarting --
Adjusting last_iter, 0, to previous model restart step.
Creating progress file: /home/i3/slots/3/progress_file_12292704.xml
last_cpu_time: 42
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3dl_2016092300_20_1018_12292704_0_r1920888991
The child process has been launched with process id: 33780
Executing the command: /home/i3/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  19:10:59 STEP    0 H=   0:00 +CPU= 20.205
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /home/i3/slots/3/ICMGGh7zg+000000
  19:11:14 STEP    1 H=   0:15 +CPU= 11.459
Moving to projects directory: /home/i3/slots/3/ICMSHh7zg+000000
  19:11:24 STEP    2 H=   0:30 +CPU=  9.753
Moving to projects directory: /home/i3/slots/3/ICMUAh7zg+000000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  19:11:45 STEP    3 H=   0:45 +CPU=  9.135
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3dl
(argv4) batchid: 1018
(argv5) wuid: 12292704
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/3
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12292704
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b23ff273b95b2d3d49265b2d6d178b19 to: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
Unzipping the namelist zip file: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
ic_ancil_file: ic_ancil_12292704
ifsdata_file: ifsdata_12292704
climate_data_file: clim_data_12292704
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/3/ic_ancil_12292704.zip
Unzipping the IC ancils zip file: /home/i3/slots/3/ic_ancil_12292704.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e1c37b5d8581da24b85a4837aa9564a8 to: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
Unzipping the ifsdata_zip file: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Unzipping the climate data zip file: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Checking for progress XML file: /home/i3/slots/3/progress_file_12292704.xml
Opened progress file ok : /home/i3/slots/3/progress_file_12292704.xml
-- Model is restarting --
Adjusting last_iter, 1, to previous model restart step.
Creating progress file: /home/i3/slots/3/progress_file_12292704.xml
last_cpu_time: 72
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3dl_2016092300_20_1018_12292704_0_r1920888991
The child process has been launched with process id: 41447
Executing the command: /home/i3/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3dl
(argv4) batchid: 1018
(argv5) wuid: 12292704
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/3
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12292704
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b23ff273b95b2d3d49265b2d6d178b19 to: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
Unzipping the namelist zip file: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
ic_ancil_file: ic_ancil_12292704
ifsdata_file: ifsdata_12292704
climate_data_file: clim_data_12292704
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/3/ic_ancil_12292704.zip
Unzipping the IC ancils zip file: /home/i3/slots/3/ic_ancil_12292704.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e1c37b5d8581da24b85a4837aa9564a8 to: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
Unzipping the ifsdata_zip file: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Unzipping the climate data zip file: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Checking for progress XML file: /home/i3/slots/3/progress_file_12292704.xml
Opened progress file ok : /home/i3/slots/3/progress_file_12292704.xml
-- Model is restarting --
Adjusting last_iter, 0, to previous model restart step.
Creating progress file: /home/i3/slots/3/progress_file_12292704.xml
last_cpu_time: 87
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3dl_2016092300_20_1018_12292704_0_r1920888991
The child process has been launched with process id: 10166
Executing the command: /home/i3/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  19:25:34 STEP    0 H=   0:00 +CPU= 14.763
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /home/i3/slots/3/ICMGGh7zg+000000
Moving to projects directory: /home/i3/slots/3/ICMSHh7zg+000000
  19:25:50 STEP    1 H=   0:15 +CPU= 13.140
Moving to projects directory: /home/i3/slots/3/ICMUAh7zg+000000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  19:26:08 STEP    2 H=   0:30 +CPU= 13.535
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  19:26:22 STEP    3 H=   0:45 +CPU= 12.455
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3dl
(argv4) batchid: 1018
(argv5) wuid: 12292704
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/3
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12292704
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b23ff273b95b2d3d49265b2d6d178b19 to: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
Unzipping the namelist zip file: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
ic_ancil_file: ic_ancil_12292704
ifsdata_file: ifsdata_12292704
climate_data_file: clim_data_12292704
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/3/ic_ancil_12292704.zip
Unzipping the IC ancils zip file: /home/i3/slots/3/ic_ancil_12292704.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e1c37b5d8581da24b85a4837aa9564a8 to: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
Unzipping the ifsdata_zip file: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Unzipping the climate data zip file: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Checking for progress XML file: /home/i3/slots/3/progress_file_12292704.xml
Opened progress file ok : /home/i3/slots/3/progress_file_12292704.xml
-- Model is restarting --
Adjusting last_iter, 4, to previous model restart step.
Creating progress file: /home/i3/slots/3/progress_file_12292704.xml
last_cpu_time: 150
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3dl_2016092300_20_1018_12292704_0_r1920888991
The child process has been launched with process id: 11674
Executing the command: /home/i3/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01

Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc
 UPDTIM MACC AEROSOL CLIMATOLOGY IS LOADED
SUECAEC: INLEV,ISYEAR,IEYEAR=  0  1957  2010
 suecaec NAERMACC =  1
 UPDT RAESHDU =  2000.0
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20160923
          90  9.999999999999998E-005
  DATE= 2016    9   23
 NORMS AT NSTEP CNT4    0
 SPECTRAL NORMS -   LOG(PREHYDS)    0.115260884514965E+02
 LEV     VORTICITY             DIVERGENCE           TEMPERATURE           KINETIC ENERGY    
 AVE 0.866154844891897E-06 0.555153323095658E-07 0.217247615887419E+03 0.180297554151816E+01
 GPNORM  HUMIDITY            AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.863401149692152E-03 0.000000000000000E+00 0.516457675471079E-02
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 NSTEP =     0 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =     0 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+000000 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+000000
 MPI-TASK:   1 -    360415332 BYTES IN      6 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.4473( 0.0%)
 NSTEP =     0 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+000000 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+000000
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0142 SECONDS 49816.8 Mbytes/s, TOTAL TIME=    0.2388( 5.9%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000000 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+000000
 MPI-TASK:   1 -   4939338816 BYTES IN    125 RECORDS TRANSFERRED IN    0.0049 SECONDS******** Mbytes/s, TOTAL TIME=    0.1145( 4.3%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000000 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000000
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0002 SECONDS     0.0 Mbytes/s, TOTAL TIME=    0.0046( 4.3%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000000 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000000
 MPI-TASK:   1 -    300128947 BYTES IN      4 RECORDS TRANSFERRED IN    0.0005 SECONDS600257.9 Mbytes/s, TOTAL TIME=    0.0056( 8.9%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =     0 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 7 lines from file: ifs.stat
 19:29:39 000000000 CNT3      -999     0.371    0.371   38.939      0:00      0:00 0.00000000000000E+00       1GB       0MB
 19:29:47 A00000000 STEPO        0     4.677    4.677   46.599      0:04      0:47 0.55515332309566E-07       2GB       0MB
 19:29:49 0AA000000 STEPO        0     0.058    0.058    2.252      0:04      0:49 0.55515332309566E-07       2GB       0MB
 19:29:50 FULLPOS-B DYNFPOS      0     0.788    0.788    1.489      0:05      0:50 0.55515332309566E-07       2GB       0MB
 19:29:55 FULLPOS-V DYNFPOS      0     2.360    2.360    4.976      0:07      0:55 0.55515332309566E-07       2GB       0MB
 19:29:56 FULLPOS-T DYNFPOS      0     0.470    0.470    0.470      0:08      0:56 0.55515332309566E-07       2GB       0MB
 19:29:56 0AAA00AAA STEPO        0     0.347    0.347    0.347      0:08      0:56 0.55515332309566E-07       2GB       0MB
------------------------------------------------
>>> Printing last 8 lines from file: /home/i3/slots/3/progress_file_12292704.xml

<running_values>
  <last_cpu_time>156.700000</last_cpu_time>
  <upload_file_number>0</upload_file_number>
  <last_iter>4</last_iter>
  <last_upload>0</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 cpdn.org