climateprediction.net home page
Task 22270039

Task 22270039

Name oifs_43r3_ps_0127_1999050100_123_968_12184771_0
Workunit 12184771
Created 22 Dec 2022, 18:00:33 UTC
Sent 18 Jan 2023, 2:36:41 UTC
Report deadline 19 Mar 2023, 2:36:41 UTC
Received 21 Jan 2023, 10:54:56 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 5 (0x00000005) Unknown error code
Computer ID 1493094
Run time 18 hours 32 min 53 sec
CPU time 17 hours 20 min 16 sec
Validate state Invalid
Credit 9,676.80
Device peak FLOPS 4.53 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,406.81 MB
Peak swap size 4,978.07 MB
Peak disk usage 3,623.44 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 5 (0x5, -251)</message>
<stderr_txt>

Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12184771.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3c16ba141c8b014d2b43511b1279a037 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12184771.xml
-- Model is restarting --
Adjusting last_iter, 2105, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
last_cpu_time: 59646
upload_file_number: 87
last_iter: 2087
last_upload: 7516800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0127_1999050100_123_968_12184771_0_r1168072986
The child process has been launched with process id: 69147
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:69147:69147] [20230119:170241:1674176561:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:04:23 STEP 2088 H=2088:00 +CPU= 41.728
  17:04:50 STEP 2089 H=2089:00 +CPU= 23.448
  17:05:15 STEP 2090 H=2090:00 +CPU= 23.689
  17:05:55 STEP 2091 H=2091:00 +CPU= 36.882
  17:06:21 STEP 2092 H=2092:00 +CPU= 24.281
  17:06:46 STEP 2093 H=2093:00 +CPU= 22.933
  17:07:28 STEP 2094 H=2094:00 +CPU= 37.995
  17:07:54 STEP 2095 H=2095:00 +CPU= 23.704
  17:08:19 STEP 2096 H=2096:00 +CPU= 23.537
  17:08:59 STEP 2097 H=2097:00 +CPU= 37.291
  17:09:24 STEP 2098 H=2098:00 +CPU= 23.472
  17:09:49 STEP 2099 H=2099:00 +CPU= 23.482
  17:10:32 STEP 2100 H=2100:00 +CPU= 39.982
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002100
  17:10:59 STEP 2101 H=2101:00 +CPU= 24.337
  17:11:25 STEP 2102 H=2102:00 +CPU= 23.738
  17:12:14 STEP 2103 H=2103:00 +CPU= 36.211
  17:12:38 STEP 2104 H=2104:00 +CPU= 23.017
  17:13:03 STEP 2105 H=2105:00 +CPU= 23.491
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0127
(argv4) batchid: 968
(argv5) wuid: 12184771
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_67eea40ba262d1408970d31af6860f95 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0127_1999050100_123_968_12184771.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0127_1999050100_123_968_12184771.zip
ic_ancil_file: ic_ancil_12184771
ifsdata_file: ifsdata_12184771
climate_data_file: clim_data_12184771
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b6094ded87981574eb53ecd246566a0b to: /var/lib/boinc-client/slots/5/ic_ancil_12184771.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12184771.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3c16ba141c8b014d2b43511b1279a037 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12184771.xml
-- Model is restarting --
Adjusting last_iter, 2106, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
last_cpu_time: 60191
upload_file_number: 87
last_iter: 2087
last_upload: 7516800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0127_1999050100_123_968_12184771_0_r1168072986
The child process has been launched with process id: 80604
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:80604:80604] [20230119:181423:1674180863:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:16:46 STEP 2088 H=2088:00 +CPU= 42.493
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0127
(argv4) batchid: 968
(argv5) wuid: 12184771
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_67eea40ba262d1408970d31af6860f95 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0127_1999050100_123_968_12184771.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0127_1999050100_123_968_12184771.zip
ic_ancil_file: ic_ancil_12184771
ifsdata_file: ifsdata_12184771
climate_data_file: clim_data_12184771
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b6094ded87981574eb53ecd246566a0b to: /var/lib/boinc-client/slots/5/ic_ancil_12184771.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12184771.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3c16ba141c8b014d2b43511b1279a037 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12184771.xml
-- Model is restarting --
Adjusting last_iter, 2089, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
last_cpu_time: 60243
upload_file_number: 87
last_iter: 2087
last_upload: 7516800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0127_1999050100_123_968_12184771_0_r1168072986
The child process has been launched with process id: 90853
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:90853:90853] [20230119:191820:1674184700:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0127
(argv4) batchid: 968
(argv5) wuid: 12184771
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_67eea40ba262d1408970d31af6860f95 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0127_1999050100_123_968_12184771.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0127_1999050100_123_968_12184771.zip
ic_ancil_file: ic_ancil_12184771
ifsdata_file: ifsdata_12184771
climate_data_file: clim_data_12184771
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b6094ded87981574eb53ecd246566a0b to: /var/lib/boinc-client/slots/5/ic_ancil_12184771.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12184771.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3c16ba141c8b014d2b43511b1279a037 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12184771.xml
-- Model is restarting --
Adjusting last_iter, 2088, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
last_cpu_time: 60274
upload_file_number: 87
last_iter: 2087
last_upload: 7516800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0127_1999050100_123_968_12184771_0_r1168072986
The child process has been launched with process id: 101035
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:101035:101035] [20230119:202248:1674188568:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0127
(argv4) batchid: 968
(argv5) wuid: 12184771
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_67eea40ba262d1408970d31af6860f95 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0127_1999050100_123_968_12184771.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0127_1999050100_123_968_12184771.zip
ic_ancil_file: ic_ancil_12184771
ifsdata_file: ifsdata_12184771
climate_data_file: clim_data_12184771
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b6094ded87981574eb53ecd246566a0b to: /var/lib/boinc-client/slots/5/ic_ancil_12184771.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12184771.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3c16ba141c8b014d2b43511b1279a037 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12184771.xml
-- Model is restarting --
Adjusting last_iter, 2088, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
last_cpu_time: 60304
upload_file_number: 87
last_iter: 2087
last_upload: 7516800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0127_1999050100_123_968_12184771_0_r1168072986
The child process has been launched with process id: 108224
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:108224:108224] [20230119:210846:1674191326:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:10:57 STEP 2088 H=2088:00 +CPU= 42.148
  21:22:04 STEP 2089 H=2089:00 +CPU= 24.156
  21:24:02 STEP 2090 H=2090:00 +CPU= 24.136
  21:24:44 STEP 2091 H=2091:00 +CPU= 35.527
  21:25:06 STEP 2092 H=2092:00 +CPU= 22.244
  21:25:29 STEP 2093 H=2093:00 +CPU= 22.239
  21:26:08 STEP 2094 H=2094:00 +CPU= 37.310
  21:26:32 STEP 2095 H=2095:00 +CPU= 22.394
  21:26:56 STEP 2096 H=2096:00 +CPU= 22.852
  21:27:33 STEP 2097 H=2097:00 +CPU= 36.713
  21:27:58 STEP 2098 H=2098:00 +CPU= 23.334
  21:28:22 STEP 2099 H=2099:00 +CPU= 23.814
  21:29:11 STEP 2100 H=2100:00 +CPU= 40.525
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002100
  21:29:34 STEP 2101 H=2101:00 +CPU= 22.745
  21:30:00 STEP 2102 H=2102:00 +CPU= 24.253
  21:30:47 STEP 2103 H=2103:00 +CPU= 37.352
  21:31:20 STEP 2104 H=2104:00 +CPU= 24.131
  21:31:44 STEP 2105 H=2105:00 +CPU= 22.386
  21:32:30 STEP 2106 H=2106:00 +CPU= 37.045
  21:32:55 STEP 2107 H=2107:00 +CPU= 23.301
  21:33:19 STEP 2108 H=2108:00 +CPU= 23.399
  21:34:00 STEP 2109 H=2109:00 +CPU= 37.212
  21:34:25 STEP 2110 H=2110:00 +CPU= 24.231
  21:34:54 STEP 2111 H=2111:00 +CPU= 23.912
  21:35:54 STEP 2112 H=2112:00 +CPU= 43.101
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0127_1999050100_123_968_12184771_0_r1168072986_87.zip
Uploading the intermediate file: upload_file_87.zip
  21:36:20 STEP 2113 H=2113:00 +CPU= 24.036
  21:36:45 STEP 2114 H=2114:00 +CPU= 24.051
  21:37:25 STEP 2115 H=2115:00 +CPU= 37.248
  21:37:50 STEP 2116 H=2116:00 +CPU= 24.144
  21:38:14 STEP 2117 H=2117:00 +CPU= 23.681
  21:38:54 STEP 2118 H=2118:00 +CPU= 37.920
  21:39:19 STEP 2119 H=2119:00 +CPU= 23.474
  21:39:45 STEP 2120 H=2120:00 +CPU= 24.548
  21:40:23 STEP 2121 H=2121:00 +CPU= 36.517
  21:40:48 STEP 2122 H=2122:00 +CPU= 24.147
  21:41:12 STEP 2123 H=2123:00 +CPU= 23.483
  21:41:58 STEP 2124 H=2124:00 +CPU= 41.060
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002124
  21:42:22 STEP 2125 H=2125:00 +CPU= 22.054
  21:42:46 STEP 2126 H=2126:00 +CPU= 23.894
  21:43:27 STEP 2127 H=2127:00 +CPU= 37.448
  21:43:52 STEP 2128 H=2128:00 +CPU= 23.066
  21:44:16 STEP 2129 H=2129:00 +CPU= 22.197
  21:44:55 STEP 2130 H=2130:00 +CPU= 37.448
  21:45:20 STEP 2131 H=2131:00 +CPU= 23.291
  21:45:45 STEP 2132 H=2132:00 +CPU= 23.622
  21:46:23 STEP 2133 H=2133:00 +CPU= 36.526
  21:46:48 STEP 2134 H=2134:00 +CPU= 23.701
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0127
(argv4) batchid: 968
(argv5) wuid: 12184771
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_67eea40ba262d1408970d31af6860f95 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0127_1999050100_123_968_12184771.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0127_1999050100_123_968_12184771.zip
ic_ancil_file: ic_ancil_12184771
ifsdata_file: ifsdata_12184771
climate_data_file: clim_data_12184771
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b6094ded87981574eb53ecd246566a0b to: /var/lib/boinc-client/slots/5/ic_ancil_12184771.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12184771.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3c16ba141c8b014d2b43511b1279a037 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12184771.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12184771.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12184771.xml
-- Model is restarting --
Adjusting last_iter, 2135, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12184771.xml
last_cpu_time: 61669
upload_file_number: 88
last_iter: 2111
last_upload: 7603200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0127_1999050100_123_968_12184771_0_r1168072986
The child process has been launched with process id: 121515
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-10:1:1:121515:121515] [20230121:015056:1674294656:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:52:21 STEP 2112 H=2112:00 +CPU= 43.253
  01:54:54 STEP 2113 H=2113:00 +CPU= 22.921
  01:55:17 STEP 2114 H=2114:00 +CPU= 22.776
  01:55:55 STEP 2115 H=2115:00 +CPU= 36.066
  01:56:18 STEP 2116 H=2116:00 +CPU= 23.348
  01:56:41 STEP 2117 H=2117:00 +CPU= 22.655
  01:57:18 STEP 2118 H=2118:00 +CPU= 36.807
  01:57:41 STEP 2119 H=2119:00 +CPU= 22.685
  01:58:04 STEP 2120 H=2120:00 +CPU= 23.326
  01:58:40 STEP 2121 H=2121:00 +CPU= 36.032
  01:59:04 STEP 2122 H=2122:00 +CPU= 23.381
  01:59:26 STEP 2123 H=2123:00 +CPU= 22.673
  02:00:08 STEP 2124 H=2124:00 +CPU= 39.660
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002124
  02:00:31 STEP 2125 H=2125:00 +CPU= 22.715
  02:00:54 STEP 2126 H=2126:00 +CPU= 23.362
  02:01:30 STEP 2127 H=2127:00 +CPU= 36.051
  02:01:54 STEP 2128 H=2128:00 +CPU= 23.366
  02:02:17 STEP 2129 H=2129:00 +CPU= 22.673
  02:02:53 STEP 2130 H=2130:00 +CPU= 36.837
  02:03:16 STEP 2131 H=2131:00 +CPU= 22.706
  02:03:40 STEP 2132 H=2132:00 +CPU= 23.342
  02:04:16 STEP 2133 H=2133:00 +CPU= 36.036
  02:04:39 STEP 2134 H=2134:00 +CPU= 23.336
  02:05:02 STEP 2135 H=2135:00 +CPU= 23.067
  02:05:57 STEP 2136 H=2136:00 +CPU= 41.809
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184771/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0127_1999050100_123_968_12184771_0_r1168072986_88.zip
Uploading the intermediate file: upload_file_88.zip
  02:06:20 STEP 2137 H=2137:00 +CPU= 22.652
free(): invalid pointer
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:27:00 1493094 22270039 oifs_43r3_ps_0127_1999050100_123_968_12184771_0 6,912,000 53,847 0.0078
09 May 2023 16:26:49 1493094 22270039 oifs_43r3_ps_0127_1999050100_123_968_12184771_0 6,048,000 47,119 0.0078
09 May 2023 16:26:35 1493094 22270039 oifs_43r3_ps_0127_1999050100_123_968_12184771_0 5,184,000 40,391 0.0078
09 May 2023 16:26:28 1493094 22270039 oifs_43r3_ps_0127_1999050100_123_968_12184771_0 4,320,000 33,665 0.0078
09 May 2023 16:26:17 1493094 22270039 oifs_43r3_ps_0127_1999050100_123_968_12184771_0 3,456,000 26,949 0.0078
09 May 2023 16:26:05 1493094 22270039 oifs_43r3_ps_0127_1999050100_123_968_12184771_0 2,592,000 20,219 0.0078
09 May 2023 16:25:54 1493094 22270039 oifs_43r3_ps_0127_1999050100_123_968_12184771_0 1,728,000 13,493 0.0078
09 May 2023 16:25:42 1493094 22270039 oifs_43r3_ps_0127_1999050100_123_968_12184771_0 864,000 6,768 0.0078


©2024 climateprediction.net