<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:25586:25586] [20230120:083630:1674200190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:25586:25586] [20230120:083630:1674200190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:25586:25586] [20230120:083630:1674200190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:25586:25586] [20230120:083630:1674200190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:25586:25586] [20230120:083630:1674200190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:25586:25586] [20230120:083630:1674200190:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
08:37:42 STEP 576 H= 576:00 +CPU= 65.338
08:38:13 STEP 577 H= 577:00 +CPU= 30.451
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0606
(argv4) batchid: 956
(argv5) wuid: 12173250
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_99523bf29fb9469e1a433b543e7d24e2 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0606_1987050100_123_956_12173250.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0606_1987050100_123_956_12173250.zip
ic_ancil_file: ic_ancil_12173250
ifsdata_file: ifsdata_12173250
climate_data_file: clim_data_12173250
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_23b33a1e71d6a89172f6fe3e5cf058d2 to: /var/lib/boinc-client/slots/5/ic_ancil_12173250.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12173250.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_00f2f9e0d2893507ed60e535f3ecd49f to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12173250.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12173250.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12173250.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12173250.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12173250.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12173250.xml
-- Model is restarting --
Adjusting last_iter, 578, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12173250.xml
last_cpu_time: 23744
upload_file_number: 24
last_iter: 575
last_upload: 2073600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875
The child process has been launched with process id: 1871
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:1871:1871] [20230120:095338:1674204818:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
09:54:48 STEP 576 H= 576:00 +CPU= 64.687
09:55:18 STEP 577 H= 577:00 +CPU= 29.762
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0606
(argv4) batchid: 956
(argv5) wuid: 12173250
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_99523bf29fb9469e1a433b543e7d24e2 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0606_1987050100_123_956_12173250.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0606_1987050100_123_956_12173250.zip
ic_ancil_file: ic_ancil_12173250
ifsdata_file: ifsdata_12173250
climate_data_file: clim_data_12173250
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_23b33a1e71d6a89172f6fe3e5cf058d2 to: /var/lib/boinc-client/slots/5/ic_ancil_12173250.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12173250.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_00f2f9e0d2893507ed60e535f3ecd49f to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12173250.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12173250.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12173250.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12173250.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12173250.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12173250.xml
-- Model is restarting --
Adjusting last_iter, 578, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12173250.xml
last_cpu_time: 23851
upload_file_number: 24
last_iter: 575
last_upload: 2073600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875
The child process has been launched with process id: 25621
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:25621:25621] [20230121:035435:1674269675:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
03:55:43 STEP 576 H= 576:00 +CPU= 62.078
03:56:13 STEP 577 H= 577:00 +CPU= 29.681
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:33:56 STEP 578 H= 578:00 +CPU= 30.945
13:34:32 STEP 579 H= 579:00 +CPU= 36.338
13:34:58 STEP 580 H= 580:00 +CPU= 26.365
13:35:28 STEP 581 H= 581:00 +CPU= 29.313
13:36:05 STEP 582 H= 582:00 +CPU= 36.955
13:36:32 STEP 583 H= 583:00 +CPU= 26.464
13:37:01 STEP 584 H= 584:00 +CPU= 29.503
13:37:37 STEP 585 H= 585:00 +CPU= 36.091
13:38:05 STEP 586 H= 586:00 +CPU= 27.681
13:38:33 STEP 587 H= 587:00 +CPU= 27.211
13:39:13 STEP 588 H= 588:00 +CPU= 40.508
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000588
13:39:43 STEP 589 H= 589:00 +CPU= 29.817
13:40:10 STEP 590 H= 590:00 +CPU= 26.409
13:40:46 STEP 591 H= 591:00 +CPU= 36.219
13:41:16 STEP 592 H= 592:00 +CPU= 29.747
13:41:39 STEP 593 H= 593:00 +CPU= 23.494
13:42:19 STEP 594 H= 594:00 +CPU= 39.161
13:42:46 STEP 595 H= 595:00 +CPU= 27.386
13:43:10 STEP 596 H= 596:00 +CPU= 23.944
13:43:51 STEP 597 H= 597:00 +CPU= 40.988
13:44:19 STEP 598 H= 598:00 +CPU= 27.192
13:44:43 STEP 599 H= 599:00 +CPU= 24.099
13:45:32 STEP 600 H= 600:00 +CPU= 48.596
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875_24.zip
Uploading the intermediate file: upload_file_24.zip
13:45:56 STEP 601 H= 601:00 +CPU= 24.050
13:46:21 STEP 602 H= 602:00 +CPU= 24.451
13:47:03 STEP 603 H= 603:00 +CPU= 42.265
13:47:27 STEP 604 H= 604:00 +CPU= 23.943
13:47:51 STEP 605 H= 605:00 +CPU= 24.274
13:48:33 STEP 606 H= 606:00 +CPU= 41.159
13:48:56 STEP 607 H= 607:00 +CPU= 23.442
13:49:26 STEP 608 H= 608:00 +CPU= 29.442
13:50:06 STEP 609 H= 609:00 +CPU= 39.671
13:50:30 STEP 610 H= 610:00 +CPU= 23.944
13:50:59 STEP 611 H= 611:00 +CPU= 29.366
13:51:46 STEP 612 H= 612:00 +CPU= 46.925
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000612
13:52:09 STEP 613 H= 613:00 +CPU= 23.335
13:52:39 STEP 614 H= 614:00 +CPU= 29.353
13:53:18 STEP 615 H= 615:00 +CPU= 38.782
13:53:42 STEP 616 H= 616:00 +CPU= 23.944
13:54:11 STEP 617 H= 617:00 +CPU= 28.796
13:54:48 STEP 618 H= 618:00 +CPU= 36.864
13:55:11 STEP 619 H= 619:00 +CPU= 23.500
13:55:41 STEP 620 H= 620:00 +CPU= 29.266
13:56:17 STEP 621 H= 621:00 +CPU= 36.199
13:56:45 STEP 622 H= 622:00 +CPU= 27.894
13:57:15 STEP 623 H= 623:00 +CPU= 29.277
13:57:59 STEP 624 H= 624:00 +CPU= 43.805
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875_25.zip
Uploading the intermediate file: upload_file_25.zip
13:58:28 STEP 625 H= 625:00 +CPU= 29.105
13:58:53 STEP 626 H= 626:00 +CPU= 24.323
13:59:31 STEP 627 H= 627:00 +CPU= 38.476
13:59:59 STEP 628 H= 628:00 +CPU= 27.759
14:00:23 STEP 629 H= 629:00 +CPU= 23.700
14:01:06 STEP 630 H= 630:00 +CPU= 42.754
14:01:29 STEP 631 H= 631:00 +CPU= 23.385
14:01:53 STEP 632 H= 632:00 +CPU= 24.124
14:02:36 STEP 633 H= 633:00 +CPU= 42.543
14:03:00 STEP 634 H= 634:00 +CPU= 24.037
14:03:24 STEP 635 H= 635:00 +CPU= 23.617
14:04:13 STEP 636 H= 636:00 +CPU= 49.243
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000636
14:04:37 STEP 637 H= 637:00 +CPU= 23.275
14:05:02 STEP 638 H= 638:00 +CPU= 25.245
14:05:45 STEP 639 H= 639:00 +CPU= 43.037
14:06:10 STEP 640 H= 640:00 +CPU= 24.321
14:06:35 STEP 641 H= 641:00 +CPU= 25.733
14:07:16 STEP 642 H= 642:00 +CPU= 40.821
14:07:40 STEP 643 H= 643:00 +CPU= 23.280
14:08:10 STEP 644 H= 644:00 +CPU= 30.240
14:08:49 STEP 645 H= 645:00 +CPU= 38.588
14:09:13 STEP 646 H= 646:00 +CPU= 24.183
14:09:44 STEP 647 H= 647:00 +CPU= 30.619
14:10:32 STEP 648 H= 648:00 +CPU= 47.866
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875_26.zip
Uploading the intermediate file: upload_file_26.zip
14:10:56 STEP 649 H= 649:00 +CPU= 24.038
14:11:27 STEP 650 H= 650:00 +CPU= 30.195
14:12:03 STEP 651 H= 651:00 +CPU= 36.385
14:12:29 STEP 652 H= 652:00 +CPU= 26.330
14:12:57 STEP 653 H= 653:00 +CPU= 27.254
14:13:34 STEP 654 H= 654:00 +CPU= 37.010
14:14:04 STEP 655 H= 655:00 +CPU= 29.962
14:14:31 STEP 656 H= 656:00 +CPU= 26.953
14:15:08 STEP 657 H= 657:00 +CPU= 36.417
14:15:37 STEP 658 H= 658:00 +CPU= 29.321
14:16:01 STEP 659 H= 659:00 +CPU= 23.315
14:16:43 STEP 660 H= 660:00 +CPU= 42.737
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000660
14:17:10 STEP 661 H= 661:00 +CPU= 26.658
14:17:34 STEP 662 H= 662:00 +CPU= 23.921
14:18:17 STEP 663 H= 663:00 +CPU= 42.713
14:18:43 STEP 664 H= 664:00 +CPU= 26.058
14:19:07 STEP 665 H= 665:00 +CPU= 23.244
14:19:54 STEP 666 H= 666:00 +CPU= 45.633
14:20:25 STEP 667 H= 667:00 +CPU= 29.098
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:21:17 STEP 668 H= 668:00 +CPU= 34.608
14:22:07 STEP 669 H= 669:00 +CPU= 46.392
14:22:46 STEP 670 H= 670:00 +CPU= 34.442
14:23:18 STEP 671 H= 671:00 +CPU= 30.016
14:24:15 STEP 672 H= 672:00 +CPU= 52.665
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875_27.zip
Uploading the intermediate file: upload_file_27.zip
14:24:55 STEP 673 H= 673:00 +CPU= 35.711
14:25:31 STEP 674 H= 674:00 +CPU= 32.211
14:26:23 STEP 675 H= 675:00 +CPU= 49.129
14:26:51 STEP 676 H= 676:00 +CPU= 26.653
14:27:30 STEP 677 H= 677:00 +CPU= 35.182
14:28:22 STEP 678 H= 678:00 +CPU= 48.793
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:30:21 STEP 679 H= 679:00 +CPU= 31.950
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:32:06 STEP 680 H= 680:00 +CPU= 35.328
14:33:03 STEP 681 H= 681:00 +CPU= 51.135
14:33:43 STEP 682 H= 682:00 +CPU= 35.474
14:34:17 STEP 683 H= 683:00 +CPU= 31.573
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0606
(argv4) batchid: 956
(argv5) wuid: 12173250
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_99523bf29fb9469e1a433b543e7d24e2 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0606_1987050100_123_956_12173250.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0606_1987050100_123_956_12173250.zip
ic_ancil_file: ic_ancil_12173250
ifsdata_file: ifsdata_12173250
climate_data_file: clim_data_12173250
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_23b33a1e71d6a89172f6fe3e5cf058d2 to: /var/lib/boinc-client/slots/5/ic_ancil_12173250.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12173250.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_00f2f9e0d2893507ed60e535f3ecd49f to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12173250.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12173250.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12173250.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12173250.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12173250.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12173250.xml
-- Model is restarting --
Adjusting last_iter, 684, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12173250.xml
last_cpu_time: 27422
upload_file_number: 28
last_iter: 671
last_upload: 2419200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875
The child process has been launched with process id: 36701
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:36701:36701] [20230121:143656:1674308216:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
14:37:57 STEP 672 H= 672:00 +CPU= 51.607
14:38:25 STEP 673 H= 673:00 +CPU= 27.886
14:38:56 STEP 674 H= 674:00 +CPU= 29.357
14:39:35 STEP 675 H= 675:00 +CPU= 37.943
14:40:06 STEP 676 H= 676:00 +CPU= 30.702
14:40:30 STEP 677 H= 677:00 +CPU= 23.903
14:41:11 STEP 678 H= 678:00 +CPU= 41.122
14:41:37 STEP 679 H= 679:00 +CPU= 26.329
14:42:02 STEP 680 H= 680:00 +CPU= 24.798
14:42:46 STEP 681 H= 681:00 +CPU= 43.432
14:43:10 STEP 682 H= 682:00 +CPU= 24.471
14:43:34 STEP 683 H= 683:00 +CPU= 24.075
14:44:21 STEP 684 H= 684:00 +CPU= 46.572
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000684
14:44:47 STEP 685 H= 685:00 +CPU= 25.581
14:45:15 STEP 686 H= 686:00 +CPU= 28.118
14:45:58 STEP 687 H= 687:00 +CPU= 42.615
14:46:23 STEP 688 H= 688:00 +CPU= 24.581
14:46:52 STEP 689 H= 689:00 +CPU= 29.282
14:47:34 STEP 690 H= 690:00 +CPU= 41.960
14:47:58 STEP 691 H= 691:00 +CPU= 23.760
14:48:30 STEP 692 H= 692:00 +CPU= 31.578
14:49:07 STEP 693 H= 693:00 +CPU= 37.596
14:49:35 STEP 694 H= 694:00 +CPU= 27.291
14:50:03 STEP 695 H= 695:00 +CPU= 28.460
14:50:50 STEP 696 H= 696:00 +CPU= 45.962
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875_28.zip
Uploading the intermediate file: upload_file_28.zip
14:51:20 STEP 697 H= 697:00 +CPU= 29.845
14:51:45 STEP 698 H= 698:00 +CPU= 24.981
14:52:28 STEP 699 H= 699:00 +CPU= 43.452
14:52:56 STEP 700 H= 700:00 +CPU= 27.618
14:53:20 STEP 701 H= 701:00 +CPU= 23.937
14:54:05 STEP 702 H= 702:00 +CPU= 44.659
14:54:29 STEP 703 H= 703:00 +CPU= 23.901
14:54:53 STEP 704 H= 704:00 +CPU= 24.614
14:55:37 STEP 705 H= 705:00 +CPU= 43.072
14:56:01 STEP 706 H= 706:00 +CPU= 24.516
14:56:27 STEP 707 H= 707:00 +CPU= 26.044
14:57:13 STEP 708 H= 708:00 +CPU= 45.756
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000708
14:57:37 STEP 709 H= 709:00 +CPU= 23.961
14:58:07 STEP 710 H= 710:00 +CPU= 29.607
14:58:45 STEP 711 H= 711:00 +CPU= 37.432
14:59:13 STEP 712 H= 712:00 +CPU= 27.956
14:59:39 STEP 713 H= 713:00 +CPU= 26.639
15:00:18 STEP 714 H= 714:00 +CPU= 38.551
15:00:47 STEP 715 H= 715:00 +CPU= 29.214
15:01:12 STEP 716 H= 716:00 +CPU= 24.582
15:01:53 STEP 717 H= 717:00 +CPU= 40.402
15:02:20 STEP 718 H= 718:00 +CPU= 27.529
15:02:46 STEP 719 H= 719:00 +CPU= 25.179
15:03:37 STEP 720 H= 720:00 +CPU= 51.506
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875_29.zip
Uploading the intermediate file: upload_file_29.zip
15:04:02 STEP 721 H= 721:00 +CPU= 24.430
15:04:28 STEP 722 H= 722:00 +CPU= 25.827
15:05:10 STEP 723 H= 723:00 +CPU= 41.792
15:05:34 STEP 724 H= 724:00 +CPU= 24.579
15:06:04 STEP 725 H= 725:00 +CPU= 29.484
15:06:46 STEP 726 H= 726:00 +CPU= 41.573
15:07:10 STEP 727 H= 727:00 +CPU= 23.926
15:07:42 STEP 728 H= 728:00 +CPU= 31.922
15:08:21 STEP 729 H= 729:00 +CPU= 39.250
15:08:46 STEP 730 H= 730:00 +CPU= 24.657
15:09:15 STEP 731 H= 731:00 +CPU= 29.216
15:09:57 STEP 732 H= 732:00 +CPU= 41.397
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000732
15:10:26 STEP 733 H= 733:00 +CPU= 28.780
15:10:52 STEP 734 H= 734:00 +CPU= 25.926
15:11:30 STEP 735 H= 735:00 +CPU= 37.966
15:12:00 STEP 736 H= 736:00 +CPU= 29.722
15:12:24 STEP 737 H= 737:00 +CPU= 24.186
15:13:06 STEP 738 H= 738:00 +CPU= 42.389
15:13:32 STEP 739 H= 739:00 +CPU= 25.135
15:13:56 STEP 740 H= 740:00 +CPU= 24.498
15:14:40 STEP 741 H= 741:00 +CPU= 43.278
15:15:05 STEP 742 H= 742:00 +CPU= 24.856
15:15:30 STEP 743 H= 743:00 +CPU= 25.165
15:16:21 STEP 744 H= 744:00 +CPU= 50.444
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875_30.zip
Uploading the intermediate file: upload_file_30.zip
15:16:45 STEP 745 H= 745:00 +CPU= 24.532
15:17:17 STEP 746 H= 746:00 +CPU= 31.619
15:17:56 STEP 747 H= 747:00 +CPU= 38.927
15:18:21 STEP 748 H= 748:00 +CPU= 24.905
15:18:53 STEP 749 H= 749:00 +CPU= 31.457
15:19:31 STEP 750 H= 750:00 +CPU= 38.477
15:19:56 STEP 751 H= 751:00 +CPU= 24.296
15:20:32 STEP 752 H= 752:00 +CPU= 33.869
15:21:24 STEP 753 H= 753:00 +CPU= 48.010
15:21:55 STEP 754 H= 754:00 +CPU= 29.143
15:22:27 STEP 755 H= 755:00 +CPU= 29.517
15:23:29 STEP 756 H= 756:00 +CPU= 55.435
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000756
15:23:57 STEP 757 H= 757:00 +CPU= 26.534
15:24:40 STEP 758 H= 758:00 +CPU= 37.701
15:25:33 STEP 759 H= 759:00 +CPU= 48.753
15:26:09 STEP 760 H= 760:00 +CPU= 32.396
15:26:41 STEP 761 H= 761:00 +CPU= 29.461
15:27:42 STEP 762 H= 762:00 +CPU= 52.596
15:28:12 STEP 763 H= 763:00 +CPU= 28.524
15:28:48 STEP 764 H= 764:00 +CPU= 33.066
15:29:39 STEP 765 H= 765:00 +CPU= 46.938
15:30:15 STEP 766 H= 766:00 +CPU= 32.404
15:30:47 STEP 767 H= 767:00 +CPU= 29.529
15:31:53 STEP 768 H= 768:00 +CPU= 60.799
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875_31.zip
Uploading the intermediate file: upload_file_31.zip
15:32:22 STEP 769 H= 769:00 +CPU= 27.366
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:33:23 STEP 770 H= 770:00 +CPU= 33.631
15:34:25 STEP 771 H= 771:00 +CPU= 53.951
15:34:58 STEP 772 H= 772:00 +CPU= 30.843
15:35:29 STEP 773 H= 773:00 +CPU= 28.768
15:36:19 STEP 774 H= 774:00 +CPU= 46.653
15:36:48 STEP 775 H= 775:00 +CPU= 27.564
15:37:22 STEP 776 H= 776:00 +CPU= 32.534
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0606
(argv4) batchid: 956
(argv5) wuid: 12173250
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_99523bf29fb9469e1a433b543e7d24e2 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0606_1987050100_123_956_12173250.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0606_1987050100_123_956_12173250.zip
ic_ancil_file: ic_ancil_12173250
ifsdata_file: ifsdata_12173250
climate_data_file: clim_data_12173250
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_23b33a1e71d6a89172f6fe3e5cf058d2 to: /var/lib/boinc-client/slots/5/ic_ancil_12173250.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12173250.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_00f2f9e0d2893507ed60e535f3ecd49f to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12173250.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12173250.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12173250.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12173250.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12173250.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12173250.xml
-- Model is restarting --
Adjusting last_iter, 776, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12173250.xml
last_cpu_time: 30918
upload_file_number: 32
last_iter: 767
last_upload: 2764800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875
The child process has been launched with process id: 38881
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:38881:38881] [20230121:153834:1674311914:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
15:39:33 STEP 768 H= 768:00 +CPU= 50.041
15:40:03 STEP 769 H= 769:00 +CPU= 28.887
15:40:42 STEP 770 H= 770:00 +CPU= 34.427
15:41:31 STEP 771 H= 771:00 +CPU= 46.207
15:42:05 STEP 772 H= 772:00 +CPU= 32.100
15:42:31 STEP 773 H= 773:00 +CPU= 25.569
15:43:23 STEP 774 H= 774:00 +CPU= 48.310
15:43:52 STEP 775 H= 775:00 +CPU= 27.086
15:44:34 STEP 776 H= 776:00 +CPU= 36.330
15:45:27 STEP 777 H= 777:00 +CPU= 47.955
15:46:12 STEP 778 H= 778:00 +CPU= 37.916
15:46:52 STEP 779 H= 779:00 +CPU= 35.117
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:48:09 STEP 780 H= 780:00 +CPU= 55.633
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:49:22 STEP 781 H= 781:00 +CPU= 35.967
15:49:53 STEP 782 H= 782:00 +CPU= 29.198
15:50:30 STEP 783 H= 783:00 +CPU= 37.345
15:51:00 STEP 784 H= 784:00 +CPU= 29.914
15:51:24 STEP 785 H= 785:00 +CPU= 24.092
15:52:04 STEP 786 H= 786:00 +CPU= 39.816
15:52:34 STEP 787 H= 787:00 +CPU= 30.259
15:52:59 STEP 788 H= 788:00 +CPU= 24.649
15:53:41 STEP 789 H= 789:00 +CPU= 41.682
15:54:08 STEP 790 H= 790:00 +CPU= 27.014
15:54:35 STEP 791 H= 791:00 +CPU= 26.677
15:55:25 STEP 792 H= 792:00 +CPU= 49.824
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173250/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0606_1987050100_123_956_12173250_1_r1810070875_32.zip
Uploading the intermediate file: upload_file_32.zip
15:55:49 STEP 793 H= 793:00 +CPU= 24.326
15:56:16 STEP 794 H= 794:00 +CPU= 27.085
15:57:00 STEP 795 H= 795:00 +CPU= 44.104
15:57:25 STEP 796 H= 796:00 +CPU= 24.655
15:57:52 STEP 797 H= 797:00 +CPU= 26.927
15:58:32 STEP 798 H= 798:00 +CPU= 40.123
15:58:56 STEP 799 H= 799:00 +CPU= 24.013
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
