<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
r 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:22809:22809] [20230225:060910:1677330550:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:22809:22809] [20230225:060910:1677330550:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:22809:22809] [20230225:060910:1677330550:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:22809:22809] [20230225:060910:1677330550:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:22809:22809] [20230225:060910:1677330550:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
06:09:55 STEP 96 H= 96:00 +CPU= 37.283
06:10:15 STEP 97 H= 97:00 +CPU= 19.511
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 01cj
(argv4) batchid: 993
(argv5) wuid: 12215185
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/41000/slots/6
Project directory is: /var/lib/41000/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185
..mkdir for temp folder for results failed
Copying: /var/lib/41000/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/41000/slots/6/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/41000/slots/6/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_205098430d18e22b0e6b9b898f00137c to: /var/lib/41000/slots/6/oifs_43r3_01cj_2019110100_123_993_12215185.zip
Unzipping the namelist zip file: /var/lib/41000/slots/6/oifs_43r3_01cj_2019110100_123_993_12215185.zip
ic_ancil_file: ic_ancil_12215185
ifsdata_file: ifsdata_12215185
climate_data_file: clim_data_12215185
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_05646b784f3149872b74865b1e4be8fd to: /var/lib/41000/slots/6/ic_ancil_12215185.zip
Unzipping the IC ancils zip file: /var/lib/41000/slots/6/ic_ancil_12215185.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ae09fe15a4e2e7a1a306b3df3c220565 to: /var/lib/41000/slots/6/ifsdata/ifsdata_12215185.zip
Unzipping the ifsdata_zip file: /var/lib/41000/slots/6/ifsdata/ifsdata_12215185.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/41000/slots/6/159l_2/clim_data_12215185.zip
Unzipping the climate data zip file: /var/lib/41000/slots/6/159l_2/clim_data_12215185.zip
Checking for progress XML file: /var/lib/41000/slots/6/progress_file_12215185.xml
Opened progress file ok : /var/lib/41000/slots/6/progress_file_12215185.xml
-- Model is restarting --
Adjusting last_iter, 98, to previous model restart step.
Creating progress file: /var/lib/41000/slots/6/progress_file_12215185.xml
last_cpu_time: 14186
upload_file_number: 4
last_iter: 97
last_upload: 345600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_01cj_2019110100_123_993_12215185_0_r1249589434
The child process has been launched with process id: 23030
Executing the command: /var/lib/41000/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:23030:23030] [20230225:061133:1677330693:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
06:12:11 STEP 96 H= 96:00 +CPU= 32.104
06:12:31 STEP 97 H= 97:00 +CPU= 19.430
06:13:00 STEP 98 H= 98:00 +CPU= 29.638
06:13:44 STEP 99 H= 99:00 +CPU= 34.180
06:14:04 STEP 100 H= 100:00 +CPU= 19.757
06:14:33 STEP 101 H= 101:00 +CPU= 28.892
06:15:04 STEP 102 H= 102:00 +CPU= 31.157
06:15:22 STEP 103 H= 103:00 +CPU= 17.232
06:15:40 STEP 104 H= 104:00 +CPU= 18.607
06:16:19 STEP 105 H= 105:00 +CPU= 36.533
06:16:43 STEP 106 H= 106:00 +CPU= 24.417
06:16:59 STEP 107 H= 107:00 +CPU= 15.935
06:18:16 STEP 108 H= 108:00 +CPU= 42.493
Moving to projects directory: /var/lib/41000/slots/6/ICMGGhnw4+000108
Moving to projects directory: /var/lib/41000/slots/6/ICMSHhnw4+000108
Moving to projects directory: /var/lib/41000/slots/6/ICMUAhnw4+000108
06:18:39 STEP 109 H= 109:00 +CPU= 18.242
06:19:08 STEP 110 H= 110:00 +CPU= 21.278
06:19:46 STEP 111 H= 111:00 +CPU= 36.538
06:20:04 STEP 112 H= 112:00 +CPU= 18.675
06:20:39 STEP 113 H= 113:00 +CPU= 30.928
06:21:31 STEP 114 H= 114:00 +CPU= 51.305
06:21:52 STEP 115 H= 115:00 +CPU= 20.891
06:22:18 STEP 116 H= 116:00 +CPU= 26.006
06:22:45 STEP 117 H= 117:00 +CPU= 26.987
06:23:04 STEP 118 H= 118:00 +CPU= 19.189
06:23:23 STEP 119 H= 119:00 +CPU= 17.970
06:24:12 STEP 120 H= 120:00 +CPU= 45.120
Moving to projects directory: /var/lib/41000/slots/6/ICMGGhnw4+000120
Moving to projects directory: /var/lib/41000/slots/6/ICMSHhnw4+000120
Moving to projects directory: /var/lib/41000/slots/6/ICMUAhnw4+000120
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMGGhnw4+000096
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMSHhnw4+000096
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMUAhnw4+000096
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMGGhnw4+000108
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMSHhnw4+000108
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMUAhnw4+000108
Zipping up the intermediate file: /var/lib/41000/projects/climateprediction.net/oifs_43r3_01cj_2019110100_123_993_12215185_0_r1249589434_4.zip
Uploading the intermediate file: upload_file_4.zip
06:24:32 STEP 121 H= 121:00 +CPU= 19.462
06:24:58 STEP 122 H= 122:00 +CPU= 25.823
06:25:34 STEP 123 H= 123:00 +CPU= 36.547
06:25:54 STEP 124 H= 124:00 +CPU= 19.435
06:26:12 STEP 125 H= 125:00 +CPU= 18.000
06:26:49 STEP 126 H= 126:00 +CPU= 36.935
06:27:21 STEP 127 H= 127:00 +CPU= 32.506
06:27:46 STEP 128 H= 128:00 +CPU= 24.454
06:28:14 STEP 129 H= 129:00 +CPU= 28.216
06:28:34 STEP 130 H= 130:00 +CPU= 20.224
06:28:50 STEP 131 H= 131:00 +CPU= 16.268
06:29:23 STEP 132 H= 132:00 +CPU= 31.265
Moving to projects directory: /var/lib/41000/slots/6/ICMGGhnw4+000132
Moving to projects directory: /var/lib/41000/slots/6/ICMSHhnw4+000132
Moving to projects directory: /var/lib/41000/slots/6/ICMUAhnw4+000132
06:29:46 STEP 133 H= 133:00 +CPU= 23.725
06:30:08 STEP 134 H= 134:00 +CPU= 21.687
06:30:36 STEP 135 H= 135:00 +CPU= 28.146
06:30:59 STEP 136 H= 136:00 +CPU= 22.383
06:31:34 STEP 137 H= 137:00 +CPU= 35.833
06:32:04 STEP 138 H= 138:00 +CPU= 29.338
06:32:23 STEP 139 H= 139:00 +CPU= 19.216
06:32:55 STEP 140 H= 140:00 +CPU= 31.780
06:33:26 STEP 141 H= 141:00 +CPU= 31.092
06:33:44 STEP 142 H= 142:00 +CPU= 17.821
06:34:04 STEP 143 H= 143:00 +CPU= 20.441
06:34:54 STEP 144 H= 144:00 +CPU= 44.131
Moving to projects directory: /var/lib/41000/slots/6/ICMGGhnw4+000144
Moving to projects directory: /var/lib/41000/slots/6/ICMSHhnw4+000144
Moving to projects directory: /var/lib/41000/slots/6/ICMUAhnw4+000144
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMGGhnw4+000120
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMSHhnw4+000120
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMUAhnw4+000120
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMGGhnw4+000132
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMSHhnw4+000132
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMUAhnw4+000132
Zipping up the intermediate file: /var/lib/41000/projects/climateprediction.net/oifs_43r3_01cj_2019110100_123_993_12215185_0_r1249589434_5.zip
Uploading the intermediate file: upload_file_5.zip
06:35:12 STEP 145 H= 145:00 +CPU= 17.864
06:35:30 STEP 146 H= 146:00 +CPU= 17.924
06:36:04 STEP 147 H= 147:00 +CPU= 33.548
06:36:22 STEP 148 H= 148:00 +CPU= 17.102
06:36:38 STEP 149 H= 149:00 +CPU= 16.412
06:37:12 STEP 150 H= 150:00 +CPU= 33.849
06:37:30 STEP 151 H= 151:00 +CPU= 17.662
06:37:54 STEP 152 H= 152:00 +CPU= 24.692
06:38:29 STEP 153 H= 153:00 +CPU= 34.565
06:38:50 STEP 154 H= 154:00 +CPU= 21.472
06:39:09 STEP 155 H= 155:00 +CPU= 19.185
06:40:05 STEP 156 H= 156:00 +CPU= 46.641
Moving to projects directory: /var/lib/41000/slots/6/ICMGGhnw4+000156
Moving to projects directory: /var/lib/41000/slots/6/ICMSHhnw4+000156
Moving to projects directory: /var/lib/41000/slots/6/ICMUAhnw4+000156
06:40:22 STEP 157 H= 157:00 +CPU= 17.035
06:40:38 STEP 158 H= 158:00 +CPU= 15.881
06:41:13 STEP 159 H= 159:00 +CPU= 27.729
06:41:36 STEP 160 H= 160:00 +CPU= 21.175
06:41:53 STEP 161 H= 161:00 +CPU= 17.650
06:42:37 STEP 162 H= 162:00 +CPU= 33.165
06:43:01 STEP 163 H= 163:00 +CPU= 22.509
06:43:23 STEP 164 H= 164:00 +CPU= 21.343
06:43:56 STEP 165 H= 165:00 +CPU= 33.701
06:44:14 STEP 166 H= 166:00 +CPU= 17.448
06:44:32 STEP 167 H= 167:00 +CPU= 17.903
06:45:32 STEP 168 H= 168:00 +CPU= 53.596
Moving to projects directory: /var/lib/41000/slots/6/ICMGGhnw4+000168
Moving to projects directory: /var/lib/41000/slots/6/ICMSHhnw4+000168
Moving to projects directory: /var/lib/41000/slots/6/ICMUAhnw4+000168
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMGGhnw4+000144
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMSHhnw4+000144
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMUAhnw4+000144
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMGGhnw4+000156
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMSHhnw4+000156
Adding to the zip: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185/ICMUAhnw4+000156
Zipping up the intermediate file: /var/lib/41000/projects/climateprediction.net/oifs_43r3_01cj_2019110100_123_993_12215185_0_r1249589434_6.zip
Uploading the intermediate file: upload_file_6.zip
06:45:53 STEP 169 H= 169:00 +CPU= 21.313
06:46:12 STEP 170 H= 170:00 +CPU= 19.359
06:46:52 STEP 171 H= 171:00 +CPU= 32.048
06:47:11 STEP 172 H= 172:00 +CPU= 18.579
06:47:28 STEP 173 H= 173:00 +CPU= 16.507
06:47:59 STEP 174 H= 174:00 +CPU= 31.281
06:48:26 STEP 175 H= 175:00 +CPU= 23.845
06:48:47 STEP 176 H= 176:00 +CPU= 20.832
06:49:18 STEP 177 H= 177:00 +CPU= 30.585
06:49:51 STEP 178 H= 178:00 +CPU= 28.417
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108070000 !!!!!!!!!!!!
06:48:26 STEP 175 H= 175:00 +CPU= 23.845
NSTEP = 176 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.087175164371E-01 1.792341320625E-01 7.104477834904E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.495516216754E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 5.58357447685786E-02 -1.80516997231861E+00 2.19691565584435E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 4.99661603926016E-02 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108071500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108073000 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108074500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108080000 !!!!!!!!!!!!
06:48:47 STEP 176 H= 176:00 +CPU= 20.832
DATE= 2019 11 8
ISTEP= 177: Reset RSOLINC= 1361.0304 RIP0= 1386.7083
SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2: 6.240006412635001E-004
9.628461857022168E-007 4.996611132045250E-007 9.607432727722499E-010
2.027017283006666E-009 7.940283000000001E-011
SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT) 20191101
10710 9.999999999999998E-005
SUECSO4: USE AVAILABLE FILES FOR DECADES: 2010 2020
SUBROUTINE SUECSO4:, UNWEIGHTED AVERAGE: 2.284115644125386E-003
NSTEP = 177 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.307050099553E-01 1.786362388479E-01 7.104756026640E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.619059613948E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 5.59236625600740E-02 -2.07687262840300E+00 1.96788642284750E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 4.94239241380323E-02 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108081500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108083000 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108084500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108090000 !!!!!!!!!!!!
06:49:18 STEP 177 H= 177:00 +CPU= 30.585
NSTEP = 178 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.241514243476E-01 1.792187661252E-01 7.138875559518E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.510892844316E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 5.44807126725809E-02 -2.00972784861366E+00 1.83376207463778E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 4.85505715615401E-02 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 01cj
(argv4) batchid: 993
(argv5) wuid: 12215185
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/41000/slots/6
Project directory is: /var/lib/41000/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185
..mkdir for temp folder for results failed
Copying: /var/lib/41000/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/41000/slots/6/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/41000/slots/6/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_205098430d18e22b0e6b9b898f00137c to: /var/lib/41000/slots/6/oifs_43r3_01cj_2019110100_123_993_12215185.zip
Unzipping the namelist zip file: /var/lib/41000/slots/6/oifs_43r3_01cj_2019110100_123_993_12215185.zip
ic_ancil_file: ic_ancil_12215185
ifsdata_file: ifsdata_12215185
climate_data_file: clim_data_12215185
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_05646b784f3149872b74865b1e4be8fd to: /var/lib/41000/slots/6/ic_ancil_12215185.zip
Unzipping the IC ancils zip file: /var/lib/41000/slots/6/ic_ancil_12215185.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ae09fe15a4e2e7a1a306b3df3c220565 to: /var/lib/41000/slots/6/ifsdata/ifsdata_12215185.zip
Unzipping the ifsdata_zip file: /var/lib/41000/slots/6/ifsdata/ifsdata_12215185.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/41000/slots/6/159l_2/clim_data_12215185.zip
Unzipping the climate data zip file: /var/lib/41000/slots/6/159l_2/clim_data_12215185.zip
Checking for progress XML file: /var/lib/41000/slots/6/progress_file_12215185.xml
Opened progress file ok : /var/lib/41000/slots/6/progress_file_12215185.xml
-- Model is restarting --
Adjusting last_iter, 179, to previous model restart step.
Creating progress file: /var/lib/41000/slots/6/progress_file_12215185.xml
last_cpu_time: 16359
upload_file_number: 7
last_iter: 169
last_upload: 604800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_01cj_2019110100_123_993_12215185_0_r1249589434
The child process has been launched with process id: 26006
Executing the command: /var/lib/41000/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26006:26006] [20230225:065116:1677333076:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
06:51:59 STEP 168 H= 168:00 +CPU= 36.550
06:52:17 STEP 169 H= 169:00 +CPU= 18.465
06:52:34 STEP 170 H= 170:00 +CPU= 16.627
06:53:11 STEP 171 H= 171:00 +CPU= 32.651
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01
NSTEP = 169 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.056710435153E-01 1.895699386068E-01 7.173859111950E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.466118017273E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 4.54464888422407E-02 -1.87107332426083E+00 2.16312713034286E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 4.26181117041635E-02 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108001500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108003000 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108004500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108010000 !!!!!!!!!!!!
06:52:17 STEP 169 H= 169:00 +CPU= 18.465
ISTEP= 170 RSOLINC= 1361.0304 RI0= 1366.0000 ZI0= 1361.0304 RIP0= 1386.5139
ISTEP= 170 ZI0= 1361.0304 ZSEASON= 1.018723689 REA=149597870000.00 RDEASO=148216723136.92
NSTEP = 170 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 4.899107627065E-01 1.885518273140E-01 7.238152509295E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.403514574044E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 3.32728500533354E-02 -1.91170156859851E+00 1.81979317033710E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 3.37285217357966E-02 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108011500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108013000 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108014500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108020000 !!!!!!!!!!!!
06:52:34 STEP 170 H= 170:00 +CPU= 16.627
DATE= 2019 11 8
ISTEP= 171: Reset RSOLINC= 1361.0304 RIP0= 1386.5417
SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2: 6.240006412635001E-004
9.628461857022168E-007 4.996611132045250E-007 9.607432727722499E-010
2.027017283006666E-009 7.940283000000001E-011
SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT) 20191101
10350 9.999999999999998E-005
SUECSO4: USE AVAILABLE FILES FOR DECADES: 2010 2020
SUBROUTINE SUECSO4:, UNWEIGHTED AVERAGE: 2.285521060502219E-003
NSTEP = 171 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.033829370610E-01 1.835781677101E-01 7.247591359831E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.527237865995E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 2.08764386271393E-02 -2.19542662572583E+00 1.93298838117403E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 2.08483463159118E-02 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108021500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108023000 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108024500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108030000 !!!!!!!!!!!!
06:53:11 STEP 171 H= 171:00 +CPU= 32.651
NSTEP = 172 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.015311781793E-01 1.798788691714E-01 7.203858245210E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.463520310252E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 1.51898538651643E-02 -2.10058458129617E+00 1.69079509356653E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 1.81982997910385E-02 -1.00000000000000E+00 1.00000000000000E+00
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 6 lines from file: ifs.stat
06:51:16 000000000 CNT3 -999 0.511 0.511 0.511 0:00 0:00 0.00000000000000E+00 1GB 0MB
06:51:23 0AAA00AAA STEPO 168 4.825 4.825 6.970 0:04 0:06 0.10742939828615E-04 1GB 0MB
06:51:59 0AAA00AAA STEPO 169 34.898 34.898 35.657 0:39 0:42 0.10742939828615E-04 2GB 0MB
06:52:17 0AAA00AAA STEPO 170 18.465 18.465 18.466 0:58 1:01 0.10742939828615E-04 2GB 0MB
06:52:34 0AAA00AAA STEPO 171 16.628 16.628 16.630 1:14 1:17 0.10742939828615E-04 2GB 0MB
06:53:11 0AAA00AAA STEPO 172 32.651 32.651 37.156 1:47 1:54 0.10742939828615E-04 2GB 0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf
CSTEP = " 168",
CTIME = "00070000 ",
NSTEPLPP = 201, 202, 49, 228226, 228227, 5*0,
IPRGPNSRES = 1,
IPRGPEWRES = 1,
IPRTRWRES = 1,
IPRTRVRES = 1,
GMASS0 = 98323.4150932735 ,
GMASSI = 98322.3896238574
/
------------------------------------------------
>>> Printing last 17 lines from file: waminfo
PARAM=10U/10V,
REPRES=GG,
ANALYSIS PERIOD = 0
FORECAST PERIOD = 10368000
WIND TIME STEP IN SECONDS = 3600
ANALYSIS FROM 20191101000000 TO 20191101000000
ASSIMILATION NO
NEW FORECAST NO
STATUS STORM1= UNFINISHED
STATUS STORM2= UNFINISHED
STATUS STORM3= UNFINISHED
DATE FOR OUTPUT OF BOTH RESTART FILES = 00000000000000
LAST DATE FOR SPECTRA FILE OUTPUT = 20200303000000
BEGIN DATE FOR USING SURFACE CURRENT = 20191101000000
PARALLEL RESTART = YES
NUMBER OF MPI TASKS USED = 1
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/41000/slots/6/progress_file_12215185.xml
<running_values>
<last_cpu_time>16480.640000</last_cpu_time>
<upload_file_number>7</upload_file_number>
<last_iter>172</last_iter>
<last_upload>604800</last_upload>
<model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 01cj
(argv4) batchid: 993
(argv5) wuid: 12215185
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/41000/slots/6
Project directory is: /var/lib/41000/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185
..mkdir for temp folder for results failed
Copying: /var/lib/41000/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/41000/slots/6/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/41000/slots/6/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_205098430d18e22b0e6b9b898f00137c to: /var/lib/41000/slots/6/oifs_43r3_01cj_2019110100_123_993_12215185.zip
Unzipping the namelist zip file: /var/lib/41000/slots/6/oifs_43r3_01cj_2019110100_123_993_12215185.zip
ic_ancil_file: ic_ancil_12215185
ifsdata_file: ifsdata_12215185
climate_data_file: clim_data_12215185
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_05646b784f3149872b74865b1e4be8fd to: /var/lib/41000/slots/6/ic_ancil_12215185.zip
Unzipping the IC ancils zip file: /var/lib/41000/slots/6/ic_ancil_12215185.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ae09fe15a4e2e7a1a306b3df3c220565 to: /var/lib/41000/slots/6/ifsdata/ifsdata_12215185.zip
Unzipping the ifsdata_zip file: /var/lib/41000/slots/6/ifsdata/ifsdata_12215185.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/41000/slots/6/159l_2/clim_data_12215185.zip
Unzipping the climate data zip file: /var/lib/41000/slots/6/159l_2/clim_data_12215185.zip
Checking for progress XML file: /var/lib/41000/slots/6/progress_file_12215185.xml
Opened progress file ok : /var/lib/41000/slots/6/progress_file_12215185.xml
-- Model is restarting --
Adjusting last_iter, 172, to previous model restart step.
Creating progress file: /var/lib/41000/slots/6/progress_file_12215185.xml
last_cpu_time: 16480
upload_file_number: 7
last_iter: 169
last_upload: 604800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_01cj_2019110100_123_993_12215185_0_r1249589434
The child process has been launched with process id: 26298
Executing the command: /var/lib/41000/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26298:26298] [20230225:065433:1677333273:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
06:55:20 STEP 168 H= 168:00 +CPU= 39.272
06:55:43 STEP 169 H= 169:00 +CPU= 22.998
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 01cj
(argv4) batchid: 993
(argv5) wuid: 12215185
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/41000/slots/6
Project directory is: /var/lib/41000/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/41000/projects/climateprediction.net/oifs_43r3_12215185
..mkdir for temp folder for results failed
Copying: /var/lib/41000/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/41000/slots/6/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/41000/slots/6/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_205098430d18e22b0e6b9b898f00137c to: /var/lib/41000/slots/6/oifs_43r3_01cj_2019110100_123_993_12215185.zip
Unzipping the namelist zip file: /var/lib/41000/slots/6/oifs_43r3_01cj_2019110100_123_993_12215185.zip
ic_ancil_file: ic_ancil_12215185
ifsdata_file: ifsdata_12215185
climate_data_file: clim_data_12215185
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_05646b784f3149872b74865b1e4be8fd to: /var/lib/41000/slots/6/ic_ancil_12215185.zip
Unzipping the IC ancils zip file: /var/lib/41000/slots/6/ic_ancil_12215185.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ae09fe15a4e2e7a1a306b3df3c220565 to: /var/lib/41000/slots/6/ifsdata/ifsdata_12215185.zip
Unzipping the ifsdata_zip file: /var/lib/41000/slots/6/ifsdata/ifsdata_12215185.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/41000/slots/6/159l_2/clim_data_12215185.zip
Unzipping the climate data zip file: /var/lib/41000/slots/6/159l_2/clim_data_12215185.zip
Checking for progress XML file: /var/lib/41000/slots/6/progress_file_12215185.xml
Opened progress file ok : /var/lib/41000/slots/6/progress_file_12215185.xml
-- Model is restarting --
Adjusting last_iter, 170, to previous model restart step.
Creating progress file: /var/lib/41000/slots/6/progress_file_12215185.xml
last_cpu_time: 16554
upload_file_number: 7
last_iter: 169
last_upload: 604800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_01cj_2019110100_123_993_12215185_0_r1249589434
The child process has been launched with process id: 26528
Executing the command: /var/lib/41000/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:m:1:1:26528:26528] [20230225:065706:1677333426:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
06:57:47 STEP 168 H= 168:00 +CPU= 34.170
06:58:07 STEP 169 H= 169:00 +CPU= 19.911
06:58:51 STEP 170 H= 170:00 +CPU= 27.515
06:59:37 STEP 171 H= 171:00 +CPU= 41.478
06:59:54 STEP 172 H= 172:00 +CPU= 17.591
07:00:18 STEP 173 H= 173:00 +CPU= 23.643
07:00:48 STEP 174 H= 174:00 +CPU= 29.579
07:01:16 STEP 175 H= 175:00 +CPU= 28.090
07:01:37 STEP 176 H= 176:00 +CPU= 21.396
07:02:34 STEP 177 H= 177:00 +CPU= 35.756
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01
ISTEP= 175 RSOLINC= 1361.0304 RI0= 1366.0000 ZI0= 1361.0304 RIP0= 1386.6528
ISTEP= 175 ZI0= 1361.0304 ZSEASON= 1.018825768 REA=149597870000.00 RDEASO=148209297810.81
NSTEP = 175 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.280448200238E-01 1.809718595933E-01 7.194885685651E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.644753027179E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 4.63608116558442E-02 -1.92804768443367E+00 2.06589622984661E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 4.29928181188428E-02 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108061500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108063000 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108064500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108070000 !!!!!!!!!!!!
07:01:16 STEP 175 H= 175:00 +CPU= 28.090
NSTEP = 176 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.087175164371E-01 1.792341320625E-01 7.104477834904E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.495516216754E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 5.58357447685786E-02 -1.80516997231861E+00 2.19691565584435E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 4.99661603926016E-02 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108071500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108073000 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108074500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108080000 !!!!!!!!!!!!
07:01:37 STEP 176 H= 176:00 +CPU= 21.396
DATE= 2019 11 8
ISTEP= 177: Reset RSOLINC= 1361.0304 RIP0= 1386.7083
SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2: 6.240006412635001E-004
9.628461857022168E-007 4.996611132045250E-007 9.607432727722499E-010
2.027017283006666E-009 7.940283000000001E-011
SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT) 20191101
10710 9.999999999999998E-005
SUECSO4: USE AVAILABLE FILES FOR DECADES: 2010 2020
SUBROUTINE SUECSO4:, UNWEIGHTED AVERAGE: 2.284115644125386E-003
NSTEP = 177 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.307050099553E-01 1.786362388479E-01 7.104756026640E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.619059613948E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 5.59236625600740E-02 -2.07687262840300E+00 1.96788642284750E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 4.94239241380323E-02 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108081500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108083000 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108084500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191108090000 !!!!!!!!!!!!
07:02:34 STEP 177 H= 177:00 +CPU= 35.756
NSTEP = 178 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.241514243476E-01 1.792187661252E-01 7.138875559518E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.510892844316E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 5.44807126725809E-02 -2.00972784861366E+00 1.83376207463778E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 4.85505715615401E-02 -1.00000000000000E+00 1.00000000000000E+00
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat
06:59:37 0AAA00AAA STEPO 172 41.477 41.477 45.143 2:06 2:31 0.10742939828615E-04 2GB 0MB
06:59:54 0AAA00AAA STEPO 173 17.591 17.591 17.596 2:23 2:48 0.10742939828615E-04 2GB 0MB
07:00:18 0AAA00AAA STEPO 174 23.802 23.802 23.807 2:47 3:12 0.10742939828615E-04 2GB 0MB
07:00:48 0AAA00AAA STEPO 175 29.420 29.420 29.425 3:17 3:41 0.10742939828615E-04 2GB 0MB
07:01:16 0AAA00AAA STEPO 176 28.090 28.090 28.392 3:45 4:10 0.10742939828615E-04 2GB 0MB
07:01:37 0AAA00AAA STEPO 177 21.397 21.397 21.427 4:06 4:31 0.10742939828615E-04 2GB 0MB
07:02:34 0AAA00AAA STEPO 178 35.756 35.756 56.783 4:42 5:28 0.10742939828615E-04 2GB 0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf
CSTEP = " 168",
CTIME = "00070000 ",
NSTEPLPP = 201, 202, 49, 228226, 228227, 5*0,
IPRGPNSRES = 1,
IPRGPEWRES = 1,
IPRTRWRES = 1,
IPRTRVRES = 1,
GMASS0 = 98323.4150932735 ,
GMASSI = 98322.3896238574
/
------------------------------------------------
>>> Printing last 17 lines from file: waminfo
PARAM=10U/10V,
REPRES=GG,
ANALYSIS PERIOD = 0
FORECAST PERIOD = 10368000
WIND TIME STEP IN SECONDS = 3600
ANALYSIS FROM 20191101000000 TO 20191101000000
ASSIMILATION NO
NEW FORECAST NO
STATUS STORM1= UNFINISHED
STATUS STORM2= UNFINISHED
STATUS STORM3= UNFINISHED
DATE FOR OUTPUT OF BOTH RESTART FILES = 00000000000000
LAST DATE FOR SPECTRA FILE OUTPUT = 20200303000000
BEGIN DATE FOR USING SURFACE CURRENT = 20191101000000
PARALLEL RESTART = YES
NUMBER OF MPI TASKS USED = 1
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/41000/slots/6/progress_file_12215185.xml
<running_values>
<last_cpu_time>16842.860000</last_cpu_time>
<upload_file_number>7</upload_file_number>
<last_iter>178</last_iter>
<last_upload>604800</last_upload>
<model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
</stderr_txt>
]]>
