<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
aborted by user</message>
<stderr_txt>
0D7B935 call_sl_ 363 call_sl.F90
oifs_43r3_model.e 0000000000D784FD call_sl_heap_ 94 call_sl_heap.F90
oifs_43r3_model.e 0000000000BF9958 gp_model_ 533 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
19:39:19 (4277): BOINC client no longer exists - exiting
19:39:19 (4277): timer handler: client dead, exiting
19:39:24 (4277): BOINC client no longer exists - exiting
19:39:24 (4277): timer handler: client dead, exiting
19:39:29 (4277): BOINC client no longer exists - exiting
19:39:29 (4277): timer handler: client dead, exiting
19:39:34 (4277): BOINC client no longer exists - exiting
19:39:34 (4277): timer handler: client dead, exiting
19:39:39 (4277): BOINC client no longer exists - exiting
19:39:39 (4277): timer handler: client dead, exiting
19:39:44 (4277): BOINC client no longer exists - exiting
19:39:44 (4277): timer handler: client dead, exiting
19:39:49 (4277): BOINC client no longer exists - exiting
19:39:49 (4277): timer handler: client dead, exiting
19:39:54 (4277): BOINC client no longer exists - exiting
19:39:54 (4277): timer handler: client dead, exiting
19:39:59 (4277): BOINC client no longer exists - exiting
19:39:59 (4277): timer handler: client dead, exiting
19:40:04 (4277): BOINC client no longer exists - exiting
19:40:04 (4277): timer handler: client dead, exiting
19:40:09 (4277): BOINC client no longer exists - exiting
19:40:09 (4277): timer handler: client dead, exiting
19:40:14 (4277): BOINC client no longer exists - exiting
19:40:14 (4277): timer handler: client dead, exiting
19:40:19 (4277): BOINC client no longer exists - exiting
19:40:19 (4277): timer handler: client dead, exiting
19:40:24 (4277): BOINC client no longer exists - exiting
19:40:24 (4277): timer handler: client dead, exiting
19:40:29 (4277): BOINC client no longer exists - exiting
19:40:29 (4277): timer handler: client dead, exiting
19:40:34 (4277): BOINC client no longer exists - exiting
19:40:34 (4277): timer handler: client dead, exiting
19:40:39 (4277): BOINC client no longer exists - exiting
19:40:39 (4277): timer handler: client dead, exiting
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0878
(argv4) batchid: 953
(argv5) wuid: 12170522
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/20
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_63a18da5ad6f3a680095fb1591534157 to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0878_1984050100_123_953_12170522.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0878_1984050100_123_953_12170522.zip
ic_ancil_file: ic_ancil_12170522
ifsdata_file: ifsdata_12170522
climate_data_file: clim_data_12170522
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0019e1a84bd4792fc63f5b0f51804864 to: /var/lib/boinc-client/slots/20/ic_ancil_12170522.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/20/ic_ancil_12170522.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_be66172c8e8e6156d75132db014ce942 to: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12170522.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12170522.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/20/159l_2/clim_data_12170522.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/20/159l_2/clim_data_12170522.zip
Checking for progress XML file: /var/lib/boinc-client/slots/20/progress_file_12170522.xml
Opened progress file ok : /var/lib/boinc-client/slots/20/progress_file_12170522.xml
-- Model is restarting --
Adjusting last_iter, 1155, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/20/progress_file_12170522.xml
last_cpu_time: 39094
upload_file_number: 48
last_iter: 1151
last_upload: 4147200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0878_1984050100_123_953_12170522_0_r968679487
The child process has been launched with process id: 2982
Executing the command: /var/lib/boinc-client/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2982:2982] [20230221:144239:1677012159:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:43:55 STEP 1152 H=1152:00 +CPU= 53.822
14:44:36 STEP 1153 H=1153:00 +CPU= 34.890
14:45:07 STEP 1154 H=1154:00 +CPU= 30.097
14:45:55 STEP 1155 H=1155:00 +CPU= 45.373
14:46:29 STEP 1156 H=1156:00 +CPU= 32.801
14:47:10 STEP 1157 H=1157:00 +CPU= 36.805
14:48:08 STEP 1158 H=1158:00 +CPU= 52.309
14:48:44 STEP 1159 H=1159:00 +CPU= 35.041
14:49:19 STEP 1160 H=1160:00 +CPU= 33.656
14:50:12 STEP 1161 H=1161:00 +CPU= 47.171
14:50:48 STEP 1162 H=1162:00 +CPU= 34.012
14:51:23 STEP 1163 H=1163:00 +CPU= 33.606
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:52:55 STEP 1164 H=1164:00 +CPU= 59.402
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+001164
double free or corruption (top)
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0878
(argv4) batchid: 953
(argv5) wuid: 12170522
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/20
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_63a18da5ad6f3a680095fb1591534157 to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0878_1984050100_123_953_12170522.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0878_1984050100_123_953_12170522.zip
ic_ancil_file: ic_ancil_12170522
ifsdata_file: ifsdata_12170522
climate_data_file: clim_data_12170522
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0019e1a84bd4792fc63f5b0f51804864 to: /var/lib/boinc-client/slots/20/ic_ancil_12170522.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/20/ic_ancil_12170522.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_be66172c8e8e6156d75132db014ce942 to: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12170522.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12170522.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/20/159l_2/clim_data_12170522.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/20/159l_2/clim_data_12170522.zip
Checking for progress XML file: /var/lib/boinc-client/slots/20/progress_file_12170522.xml
Opened progress file ok : /var/lib/boinc-client/slots/20/progress_file_12170522.xml
-- Model is restarting --
Adjusting last_iter, 1165, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/20/progress_file_12170522.xml
last_cpu_time: 39634
upload_file_number: 48
last_iter: 1151
last_upload: 4147200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0878_1984050100_123_953_12170522_0_r968679487
The child process has been launched with process id: 2903
Executing the command: /var/lib/boinc-client/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223055:1677040255:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
22:31:58 STEP 1152 H=1152:00 +CPU= 53.242
22:32:34 STEP 1153 H=1153:00 +CPU= 33.076
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:33:18 STEP 1154 H=1154:00 +CPU= 31.021
22:34:04 STEP 1155 H=1155:00 +CPU= 44.742
22:34:34 STEP 1156 H=1156:00 +CPU= 29.037
22:35:05 STEP 1157 H=1157:00 +CPU= 30.331
22:35:49 STEP 1158 H=1158:00 +CPU= 43.225
22:36:22 STEP 1159 H=1159:00 +CPU= 31.789
22:36:56 STEP 1160 H=1160:00 +CPU= 33.251
22:37:39 STEP 1161 H=1161:00 +CPU= 43.138
22:38:10 STEP 1162 H=1162:00 +CPU= 30.241
22:38:46 STEP 1163 H=1163:00 +CPU= 34.876
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223918:1677040758:503.150] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4415MB (heap), 4304MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223918:1677040758:503.165] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223918:1677040758:503.165] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223918:1677040758:503.165] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223918:1677040758:503.200] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223918:1677040758:503.200] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4415 MB (maxheap), 4304 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.248] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.255] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.261] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.261] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.268] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.268] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.268] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.268] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.268] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.268] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.268] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADDRV
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.268] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADINTG
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223919:1677040759:503.268] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-RADINTG-INPUT LOOP2 (1206)
double free or corruption (top)
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223920:1677040760:504.268] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Gary:1:1:2903:2903] [20230221:223920:1677040760:504.268] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 00000000012BD2AB radintg_ 945 radintg.F90
oifs_43r3_model.e 0000000001270124 raddrv_ 537 raddrv.F90
oifs_43r3_model.e 0000000000BFD839 gp_model_ 482 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
22:39:23 (2860): BOINC client no longer exists - exiting
22:39:23 (2860): timer handler: client dead, exiting
22:39:28 (2860): BOINC client no longer exists - exiting
22:39:28 (2860): timer handler: client dead, exiting
22:39:33 (2860): BOINC client no longer exists - exiting
22:39:33 (2860): timer handler: client dead, exiting
22:39:38 (2860): BOINC client no longer exists - exiting
22:39:38 (2860): timer handler: client dead, exiting
22:39:43 (2860): BOINC client no longer exists - exiting
22:39:43 (2860): timer handler: client dead, exiting
22:39:48 (2860): BOINC client no longer exists - exiting
22:39:48 (2860): timer handler: client dead, exiting
22:39:53 (2860): BOINC client no longer exists - exiting
22:39:53 (2860): timer handler: client dead, exiting
22:39:58 (2860): BOINC client no longer exists - exiting
22:39:58 (2860): timer handler: client dead, exiting
22:40:03 (2860): BOINC client no longer exists - exiting
22:40:03 (2860): timer handler: client dead, exiting
22:40:08 (2860): BOINC client no longer exists - exiting
22:40:08 (2860): timer handler: client dead, exiting
22:40:13 (2860): BOINC client no longer exists - exiting
22:40:13 (2860): timer handler: client dead, exiting
22:40:18 (2860): BOINC client no longer exists - exiting
22:40:18 (2860): timer handler: client dead, exiting
22:40:23 (2860): BOINC client no longer exists - exiting
22:40:23 (2860): timer handler: client dead, exiting
22:40:28 (2860): BOINC client no longer exists - exiting
22:40:28 (2860): timer handler: client dead, exiting
22:40:33 (2860): BOINC client no longer exists - exiting
22:40:33 (2860): timer handler: client dead, exiting
22:40:38 (2860): BOINC client no longer exists - exiting
22:40:38 (2860): timer handler: client dead, exiting
22:40:43 (2860): BOINC client no longer exists - exiting
22:40:43 (2860): timer handler: client dead, exiting
22:40:48 (2860): BOINC client no longer exists - exiting
22:40:48 (2860): timer handler: client dead, exiting
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0878
(argv4) batchid: 953
(argv5) wuid: 12170522
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/20
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_63a18da5ad6f3a680095fb1591534157 to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0878_1984050100_123_953_12170522.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0878_1984050100_123_953_12170522.zip
ic_ancil_file: ic_ancil_12170522
ifsdata_file: ifsdata_12170522
climate_data_file: clim_data_12170522
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0019e1a84bd4792fc63f5b0f51804864 to: /var/lib/boinc-client/slots/20/ic_ancil_12170522.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/20/ic_ancil_12170522.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_be66172c8e8e6156d75132db014ce942 to: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12170522.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12170522.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/20/159l_2/clim_data_12170522.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/20/159l_2/clim_data_12170522.zip
Checking for progress XML file: /var/lib/boinc-client/slots/20/progress_file_12170522.xml
Opened progress file ok : /var/lib/boinc-client/slots/20/progress_file_12170522.xml
-- Model is restarting --
Adjusting last_iter, 1164, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/20/progress_file_12170522.xml
last_cpu_time: 40090
upload_file_number: 48
last_iter: 1151
last_upload: 4147200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0878_1984050100_123_953_12170522_0_r968679487
The child process has been launched with process id: 2940
Executing the command: /var/lib/boinc-client/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:141801:1677097081:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
14:19:11 STEP 1152 H=1152:00 +CPU= 54.943
14:19:47 STEP 1153 H=1153:00 +CPU= 34.052
14:20:19 STEP 1154 H=1154:00 +CPU= 30.588
14:21:03 STEP 1155 H=1155:00 +CPU= 43.722
14:21:36 STEP 1156 H=1156:00 +CPU= 31.988
14:22:09 STEP 1157 H=1157:00 +CPU= 32.157
14:22:53 STEP 1158 H=1158:00 +CPU= 43.613
14:23:26 STEP 1159 H=1159:00 +CPU= 31.889
14:23:59 STEP 1160 H=1160:00 +CPU= 32.720
14:24:43 STEP 1161 H=1161:00 +CPU= 42.850
14:25:16 STEP 1162 H=1162:00 +CPU= 32.514
14:25:49 STEP 1163 H=1163:00 +CPU= 32.285
14:26:35 STEP 1164 H=1164:00 +CPU= 45.283
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+001164
14:27:08 STEP 1165 H=1165:00 +CPU= 32.220
14:27:39 STEP 1166 H=1166:00 +CPU= 30.041
14:28:23 STEP 1167 H=1167:00 +CPU= 43.665
14:28:53 STEP 1168 H=1168:00 +CPU= 29.898
14:29:25 STEP 1169 H=1169:00 +CPU= 31.517
14:30:05 STEP 1170 H=1170:00 +CPU= 40.186
14:30:32 STEP 1171 H=1171:00 +CPU= 26.609
14:31:03 STEP 1172 H=1172:00 +CPU= 30.955
14:31:45 STEP 1173 H=1173:00 +CPU= 41.095
14:32:14 STEP 1174 H=1174:00 +CPU= 28.880
14:32:46 STEP 1175 H=1175:00 +CPU= 32.347
14:33:34 STEP 1176 H=1176:00 +CPU= 46.224
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0878_1984050100_123_953_12170522_0_r968679487_48.zip
Uploading the intermediate file: upload_file_48.zip
14:34:02 STEP 1177 H=1177:00 +CPU= 28.221
14:34:33 STEP 1178 H=1178:00 +CPU= 30.770
14:35:15 STEP 1179 H=1179:00 +CPU= 41.475
14:35:44 STEP 1180 H=1180:00 +CPU= 28.204
14:36:17 STEP 1181 H=1181:00 +CPU= 32.448
14:36:56 STEP 1182 H=1182:00 +CPU= 39.363
14:37:25 STEP 1183 H=1183:00 +CPU= 28.486
14:38:00 STEP 1184 H=1184:00 +CPU= 32.469
14:38:39 STEP 1185 H=1185:00 +CPU= 38.094
14:39:10 STEP 1186 H=1186:00 +CPU= 30.574
14:39:39 STEP 1187 H=1187:00 +CPU= 28.915
14:40:29 STEP 1188 H=1188:00 +CPU= 48.384
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+001188
14:41:01 STEP 1189 H=1189:00 +CPU= 30.805
14:41:32 STEP 1190 H=1190:00 +CPU= 30.636
14:42:14 STEP 1191 H=1191:00 +CPU= 41.761
14:42:44 STEP 1192 H=1192:00 +CPU= 30.336
14:43:16 STEP 1193 H=1193:00 +CPU= 31.623
14:44:00 STEP 1194 H=1194:00 +CPU= 43.191
14:44:29 STEP 1195 H=1195:00 +CPU= 28.988
14:45:02 STEP 1196 H=1196:00 +CPU= 32.117
14:45:41 STEP 1197 H=1197:00 +CPU= 38.661
14:46:09 STEP 1198 H=1198:00 +CPU= 28.391
14:46:43 STEP 1199 H=1199:00 +CPU= 32.954
14:47:36 STEP 1200 H=1200:00 +CPU= 52.701
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0878_1984050100_123_953_12170522_0_r968679487_49.zip
Uploading the intermediate file: upload_file_49.zip
14:48:06 STEP 1201 H=1201:00 +CPU= 28.940
14:48:38 STEP 1202 H=1202:00 +CPU= 32.175
14:49:21 STEP 1203 H=1203:00 +CPU= 42.549
14:49:52 STEP 1204 H=1204:00 +CPU= 30.319
14:50:25 STEP 1205 H=1205:00 +CPU= 32.771
14:51:04 STEP 1206 H=1206:00 +CPU= 38.733
14:51:31 STEP 1207 H=1207:00 +CPU= 27.022
14:52:04 STEP 1208 H=1208:00 +CPU= 32.484
14:52:49 STEP 1209 H=1209:00 +CPU= 44.145
14:53:19 STEP 1210 H=1210:00 +CPU= 29.881
14:53:52 STEP 1211 H=1211:00 +CPU= 32.782
14:54:36 STEP 1212 H=1212:00 +CPU= 43.767
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+001212
14:55:05 STEP 1213 H=1213:00 +CPU= 28.431
14:55:38 STEP 1214 H=1214:00 +CPU= 32.708
14:56:19 STEP 1215 H=1215:00 +CPU= 40.601
14:56:51 STEP 1216 H=1216:00 +CPU= 30.871
14:57:23 STEP 1217 H=1217:00 +CPU= 31.848
14:58:07 STEP 1218 H=1218:00 +CPU= 42.980
14:58:37 STEP 1219 H=1219:00 +CPU= 29.512
14:59:09 STEP 1220 H=1220:00 +CPU= 31.888
14:59:51 STEP 1221 H=1221:00 +CPU= 41.740
15:00:22 STEP 1222 H=1222:00 +CPU= 30.479
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:02:55 STEP 1223 H=1223:00 +CPU= 46.863
15:03:42 STEP 1224 H=1224:00 +CPU= 46.308
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0878_1984050100_123_953_12170522_0_r968679487_50.zip
Uploading the intermediate file: upload_file_50.zip
15:04:16 STEP 1225 H=1225:00 +CPU= 32.876
15:04:48 STEP 1226 H=1226:00 +CPU= 31.857
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.711] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4289MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.712] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.712] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.712] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.714] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.714] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4289 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS CLDPP T/S (1002)
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALLPAR
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CLOUD_LAYER
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150528:1677099928:2846.717] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CLOUDSC
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150529:1677099929:2847.717] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Gary:1:1:2940:2940] [20230222:150529:1677099929:2847.717] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001E3F820 Unknown Unknown Unknown
oifs_43r3_model.e 000000000103278A cloudsc_ 2942 cloudsc.F90
oifs_43r3_model.e 0000000001014D57 cloud_layer_ 131 cloud_layer.F90
oifs_43r3_model.e 0000000000F1A790 callpar_ 1223 callpar.F90
oifs_43r3_model.e 0000000000E4940F ec_phys_ 670 ec_phys.F90
oifs_43r3_model.e 0000000000E3610C ec_phys_drv_ 599 ec_phys_drv.F90
oifs_43r3_model.e 0000000000BFA3D3 gp_model_ 613 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0878
(argv4) batchid: 953
(argv5) wuid: 12170522
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/20
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170522
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_63a18da5ad6f3a680095fb1591534157 to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0878_1984050100_123_953_12170522.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0878_1984050100_123_953_12170522.zip
ic_ancil_file: ic_ancil_12170522
ifsdata_file: ifsdata_12170522
climate_data_file: clim_data_12170522
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0019e1a84bd4792fc63f5b0f51804864 to: /var/lib/boinc-client/slots/20/ic_ancil_12170522.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/20/ic_ancil_12170522.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_be66172c8e8e6156d75132db014ce942 to: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12170522.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12170522.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/20/159l_2/clim_data_12170522.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/20/159l_2/clim_data_12170522.zip
Checking for progress XML file: /var/lib/boinc-client/slots/20/progress_file_12170522.xml
Opened progress file ok : /var/lib/boinc-client/slots/20/progress_file_12170522.xml
-- Model is restarting --
Adjusting last_iter, 1227, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/20/progress_file_12170522.xml
last_cpu_time: 42767
upload_file_number: 51
last_iter: 1223
last_upload: 4406400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0878_1984050100_123_953_12170522_0_r968679487
The child process has been launched with process id: 2889
Executing the command: /var/lib/boinc-client/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Gary:1:1:2889:2889] [20230303:221419:1677903259:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
22:15:22 STEP 1224 H=1224:00 +CPU= 50.765
22:15:54 STEP 1225 H=1225:00 +CPU= 29.398
22:16:26 STEP 1226 H=1226:00 +CPU= 31.816
22:17:06 STEP 1227 H=1227:00 +CPU= 39.178
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
