<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:5292:5292] [20221222:222926:1671748166:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:5292:5292] [20221222:222926:1671748166:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:31:59 STEP 624 H= 624:00 +CPU= 34.152
22:33:18 STEP 625 H= 625:00 +CPU= 19.850
22:34:05 STEP 626 H= 626:00 +CPU= 19.827
22:35:33 STEP 627 H= 627:00 +CPU= 30.245
22:38:20 STEP 628 H= 628:00 +CPU= 22.037
22:41:22 STEP 629 H= 629:00 +CPU= 22.832
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:46:22 STEP 630 H= 630:00 +CPU= 36.179
double free or corruption (top)
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1986050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0279
(argv4) batchid: 955
(argv5) wuid: 12171923
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_84b84ef377bf4ababbaae97b64ae025e to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0279_1986050100_123_955_12171923.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0279_1986050100_123_955_12171923.zip
ic_ancil_file: ic_ancil_12171923
ifsdata_file: ifsdata_12171923
climate_data_file: clim_data_12171923
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_45d490116ddba3f50f21f9e45b389337 to: /var/lib/boinc-client/slots/5/ic_ancil_12171923.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12171923.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2f55adac371daef366a8fdaf611d7d1b to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12171923.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12171923.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12171923.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12171923.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12171923.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12171923.xml
-- Model is restarting --
Adjusting last_iter, 631, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12171923.xml
last_cpu_time: 15115
upload_file_number: 26
last_iter: 623
last_upload: 2246400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0279_1986050100_123_955_12171923_0_r1454888475
The child process has been launched with process id: 87481
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:143748:1671806268:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
14:39:27 STEP 624 H= 624:00 +CPU= 35.220
14:40:18 STEP 625 H= 625:00 +CPU= 19.721
14:40:36 STEP 626 H= 626:00 +CPU= 18.046
14:41:05 STEP 627 H= 627:00 +CPU= 27.918
14:41:29 STEP 628 H= 628:00 +CPU= 19.843
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.861] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3993MB (heap), 4090MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.866] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.867] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.867] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.877] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.877] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3993 MB (maxheap), 4090 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.897] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.899] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.901] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.901] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.902] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.902] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.902] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.902] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.902] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.902] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.902] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.902] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS (1001)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.902] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144804:1671806884:615.902] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALLPAR
double free or corruption (top)
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144805:1671806885:616.902] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:homeserver1:1:1:87481:87481] [20221223:144805:1671806885:616.902] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000402D30 yomhook_mp_dr_hoo 69 yomhook.F90
oifs_43r3_model.e 000000000124CF60 local_arrays_fin_ 62 local_arrays_fin.F90
oifs_43r3_model.e 0000000000F2BF4F callpar_ 1745 callpar.F90
oifs_43r3_model.e 0000000000E4940F ec_phys_ 670 ec_phys.F90
oifs_43r3_model.e 0000000000E3610C ec_phys_drv_ 599 ec_phys_drv.F90
oifs_43r3_model.e 0000000000BFA3D3 gp_model_ 613 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1986050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0279
(argv4) batchid: 955
(argv5) wuid: 12171923
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_84b84ef377bf4ababbaae97b64ae025e to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0279_1986050100_123_955_12171923.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0279_1986050100_123_955_12171923.zip
ic_ancil_file: ic_ancil_12171923
ifsdata_file: ifsdata_12171923
climate_data_file: clim_data_12171923
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_45d490116ddba3f50f21f9e45b389337 to: /var/lib/boinc-client/slots/5/ic_ancil_12171923.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12171923.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2f55adac371daef366a8fdaf611d7d1b to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12171923.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12171923.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12171923.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12171923.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12171923.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12171923.xml
-- Model is restarting --
Adjusting last_iter, 629, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12171923.xml
last_cpu_time: 15246
upload_file_number: 26
last_iter: 623
last_upload: 2246400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0279_1986050100_123_955_12171923_0_r1454888475
The child process has been launched with process id: 2103
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:2103:2103] [20221227:231257:1672182777:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:14:41 STEP 624 H= 624:00 +CPU= 34.804
23:15:24 STEP 625 H= 625:00 +CPU= 18.749
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1986050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0279
(argv4) batchid: 955
(argv5) wuid: 12171923
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_84b84ef377bf4ababbaae97b64ae025e to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0279_1986050100_123_955_12171923.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0279_1986050100_123_955_12171923.zip
ic_ancil_file: ic_ancil_12171923
ifsdata_file: ifsdata_12171923
climate_data_file: clim_data_12171923
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_45d490116ddba3f50f21f9e45b389337 to: /var/lib/boinc-client/slots/5/ic_ancil_12171923.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12171923.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2f55adac371daef366a8fdaf611d7d1b to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12171923.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12171923.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12171923.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12171923.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12171923.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12171923.xml
-- Model is restarting --
Adjusting last_iter, 625, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12171923.xml
last_cpu_time: 15297
upload_file_number: 26
last_iter: 623
last_upload: 2246400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0279_1986050100_123_955_12171923_0_r1454888475
The child process has been launched with process id: 4050
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:4050:4050] [20221227:231634:1672182994:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
23:17:12 STEP 624 H= 624:00 +CPU= 32.357
23:17:40 STEP 625 H= 625:00 +CPU= 18.458
23:18:22 STEP 626 H= 626:00 +CPU= 19.098
23:18:51 STEP 627 H= 627:00 +CPU= 28.813
23:19:10 STEP 628 H= 628:00 +CPU= 18.359
23:19:29 STEP 629 H= 629:00 +CPU= 19.126
23:20:25 STEP 630 H= 630:00 +CPU= 29.779
23:20:43 STEP 631 H= 631:00 +CPU= 17.995
23:21:03 STEP 632 H= 632:00 +CPU= 18.878
23:22:14 STEP 633 H= 633:00 +CPU= 30.339
23:22:33 STEP 634 H= 634:00 +CPU= 18.994
23:22:51 STEP 635 H= 635:00 +CPU= 18.012
23:23:55 STEP 636 H= 636:00 +CPU= 32.786
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000636
23:24:13 STEP 637 H= 637:00 +CPU= 18.238
23:24:32 STEP 638 H= 638:00 +CPU= 18.648
23:25:22 STEP 639 H= 639:00 +CPU= 29.807
23:25:41 STEP 640 H= 640:00 +CPU= 18.797
23:25:59 STEP 641 H= 641:00 +CPU= 18.165
23:26:49 STEP 642 H= 642:00 +CPU= 30.705
23:27:07 STEP 643 H= 643:00 +CPU= 18.457
23:27:26 STEP 644 H= 644:00 +CPU= 18.710
23:27:57 STEP 645 H= 645:00 +CPU= 29.368
23:28:41 STEP 646 H= 646:00 +CPU= 19.503
23:29:00 STEP 647 H= 647:00 +CPU= 18.679
23:30:07 STEP 648 H= 648:00 +CPU= 34.540
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0279_1986050100_123_955_12171923_0_r1454888475_26.zip
Uploading the intermediate file: upload_file_26.zip
23:30:25 STEP 649 H= 649:00 +CPU= 18.267
23:30:44 STEP 650 H= 650:00 +CPU= 18.594
23:31:48 STEP 651 H= 651:00 +CPU= 29.886
23:32:07 STEP 652 H= 652:00 +CPU= 18.663
23:32:25 STEP 653 H= 653:00 +CPU= 18.111
23:32:56 STEP 654 H= 654:00 +CPU= 29.753
23:33:28 STEP 655 H= 655:00 +CPU= 19.095
23:33:47 STEP 656 H= 656:00 +CPU= 18.824
23:34:16 STEP 657 H= 657:00 +CPU= 28.569
23:34:58 STEP 658 H= 658:00 +CPU= 19.656
23:35:17 STEP 659 H= 659:00 +CPU= 18.236
23:35:49 STEP 660 H= 660:00 +CPU= 31.669
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000660
23:36:45 STEP 661 H= 661:00 +CPU= 19.667
23:37:04 STEP 662 H= 662:00 +CPU= 18.672
23:37:33 STEP 663 H= 663:00 +CPU= 28.482
23:37:53 STEP 664 H= 664:00 +CPU= 19.770
23:38:26 STEP 665 H= 665:00 +CPU= 18.706
23:38:56 STEP 666 H= 666:00 +CPU= 29.365
23:39:14 STEP 667 H= 667:00 +CPU= 18.253
23:39:57 STEP 668 H= 668:00 +CPU= 19.832
23:40:26 STEP 669 H= 669:00 +CPU= 27.589
23:40:44 STEP 670 H= 670:00 +CPU= 18.190
23:41:04 STEP 671 H= 671:00 +CPU= 19.509
23:42:21 STEP 672 H= 672:00 +CPU= 34.205
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0279_1986050100_123_955_12171923_0_r1454888475_27.zip
Uploading the intermediate file: upload_file_27.zip
23:42:39 STEP 673 H= 673:00 +CPU= 17.966
23:43:40 STEP 674 H= 674:00 +CPU= 19.798
23:44:08 STEP 675 H= 675:00 +CPU= 28.566
23:44:27 STEP 676 H= 676:00 +CPU= 18.652
23:44:49 STEP 677 H= 677:00 +CPU= 18.619
23:45:59 STEP 678 H= 678:00 +CPU= 30.105
23:46:17 STEP 679 H= 679:00 +CPU= 18.231
23:46:36 STEP 680 H= 680:00 +CPU= 18.632
23:47:40 STEP 681 H= 681:00 +CPU= 29.894
23:47:59 STEP 682 H= 682:00 +CPU= 18.801
23:48:17 STEP 683 H= 683:00 +CPU= 18.096
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:53:27 STEP 684 H= 684:00 +CPU= 35.933
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000684
23:53:48 STEP 685 H= 685:00 +CPU= 18.290
23:54:07 STEP 686 H= 686:00 +CPU= 18.694
23:55:11 STEP 687 H= 687:00 +CPU= 29.973
23:55:30 STEP 688 H= 688:00 +CPU= 18.728
23:55:48 STEP 689 H= 689:00 +CPU= 18.180
23:57:04 STEP 690 H= 690:00 +CPU= 32.307
23:57:23 STEP 691 H= 691:00 +CPU= 18.335
23:57:42 STEP 692 H= 692:00 +CPU= 18.855
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:59:45 STEP 693 H= 693:00 +CPU= 31.163
00:00:06 STEP 694 H= 694:00 +CPU= 19.136
00:00:26 STEP 695 H= 695:00 +CPU= 18.880
00:02:09 STEP 696 H= 696:00 +CPU= 35.010
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0279_1986050100_123_955_12171923_0_r1454888475_28.zip
Uploading the intermediate file: upload_file_28.zip
00:02:29 STEP 697 H= 697:00 +CPU= 18.100
00:03:50 STEP 698 H= 698:00 +CPU= 20.616
00:04:19 STEP 699 H= 699:00 +CPU= 28.805
00:04:38 STEP 700 H= 700:00 +CPU= 18.652
00:05:28 STEP 701 H= 701:00 +CPU= 19.297
00:05:58 STEP 702 H= 702:00 +CPU= 29.082
00:06:16 STEP 703 H= 703:00 +CPU= 17.876
00:07:12 STEP 704 H= 704:00 +CPU= 19.428
00:07:40 STEP 705 H= 705:00 +CPU= 28.094
00:07:59 STEP 706 H= 706:00 +CPU= 18.485
00:08:44 STEP 707 H= 707:00 +CPU= 19.887
00:09:16 STEP 708 H= 708:00 +CPU= 31.351
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000708
00:09:35 STEP 709 H= 709:00 +CPU= 18.322
00:11:02 STEP 710 H= 710:00 +CPU= 20.873
00:11:30 STEP 711 H= 711:00 +CPU= 28.330
00:12:55 STEP 712 H= 712:00 +CPU= 21.456
00:13:14 STEP 713 H= 713:00 +CPU= 17.684
00:13:43 STEP 714 H= 714:00 +CPU= 28.494
00:14:01 STEP 715 H= 715:00 +CPU= 17.689
00:14:52 STEP 716 H= 716:00 +CPU= 19.989
00:15:21 STEP 717 H= 717:00 +CPU= 28.345
00:15:40 STEP 718 H= 718:00 +CPU= 18.267
00:15:59 STEP 719 H= 719:00 +CPU= 18.742
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1986050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0279
(argv4) batchid: 955
(argv5) wuid: 12171923
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12171923
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_84b84ef377bf4ababbaae97b64ae025e to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0279_1986050100_123_955_12171923.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0279_1986050100_123_955_12171923.zip
ic_ancil_file: ic_ancil_12171923
ifsdata_file: ifsdata_12171923
climate_data_file: clim_data_12171923
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_45d490116ddba3f50f21f9e45b389337 to: /var/lib/boinc-client/slots/5/ic_ancil_12171923.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12171923.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2f55adac371daef366a8fdaf611d7d1b to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12171923.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12171923.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12171923.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12171923.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12171923.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12171923.xml
-- Model is restarting --
Adjusting last_iter, 720, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12171923.xml
last_cpu_time: 17477
upload_file_number: 29
last_iter: 719
last_upload: 2505600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0279_1986050100_123_955_12171923_0_r1454888475
The child process has been launched with process id: 8502
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001745:1672186665:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
ABORT! 1 !! *** WAVE MODEL HAS ABORTED
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.941] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3326 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.941] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.941] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.941] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.941] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.941] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.941] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.941] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WVCOUPLE
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.941] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAVEMDL
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.941] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : INITMDL
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.941] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : USERIN
SDL_TRACEBACK: Calling INTEL_TRBK, THRD = 1
Process 1 thread 1 calling linux_trbk from intel_trbk()
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/5/oifs_43r3_model.exe' : sigcontextptr=0x7ffeb7d20520
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/5/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
SDL_TRACEBACK: Done INTEL_TRBK, THRD = 1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.981] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3326MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.981] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.981] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.981] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x7400002136], nsigs = 1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001900:1672186740:74.981] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001901:1672186741:75.982] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:homeserver1:1:1:8502:8502] [20221228:001901:1672186741:75.982] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC903B Unknown Unknown Unknown
oifs_43r3_model.e 00000000008C2880 sdl_mod_mp_sdl_sr 100 sdl_mod.F90
oifs_43r3_model.e 000000000081D8AF abor1_ 41 abor1.F90
oifs_43r3_model.e 00000000015F8ACA mpabort_ 50 mpabort.F
oifs_43r3_model.e 00000000015D51C8 abort1_ 44 abort1.F
oifs_43r3_model.e 000000000166B07F userin_ 553 userin.F
oifs_43r3_model.e 00000000015D7FF0 initmdl_ 452 initmdl.F
oifs_43r3_model.e 00000000008711E0 wavemdl_ 393 wavemdl.F
oifs_43r3_model.e 00000000007E0165 wvcouple_ 260 wvcouple.F90
oifs_43r3_model.e 000000000055E4AD cnt4_ 1283 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully
</stderr_txt>
]]>
