climateprediction.net home page
Task 22253270

Task 22253270

Name oifs_43r3_ps_0348_1983050100_123_952_12168992_0
Workunit 12168992
Created 21 Dec 2022, 0:32:51 UTC
Sent 13 Jan 2023, 15:54:53 UTC
Report deadline 12 Feb 2023, 15:54:53 UTC
Received 14 Jan 2023, 23:11:57 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1463165
Run time 16 hours 41 min 58 sec
CPU time 16 hours 40 min 43 sec
Validate state Invalid
Credit 6,048.00
Device peak FLOPS 4.74 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,794.29 MB
Peak swap size 4,978.10 MB
Peak disk usage 7,716.91 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 2117 H=2117:00 +CPU= 18.920
  08:12:52 STEP 2118 H=2118:00 +CPU= 31.306
  08:13:11 STEP 2119 H=2119:00 +CPU= 18.916
  08:13:30 STEP 2120 H=2120:00 +CPU= 19.523
  08:14:01 STEP 2121 H=2121:00 +CPU= 30.553
  08:14:20 STEP 2122 H=2122:00 +CPU= 19.536
  08:14:39 STEP 2123 H=2123:00 +CPU= 18.913
  08:15:13 STEP 2124 H=2124:00 +CPU= 33.763
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002124
  08:15:32 STEP 2125 H=2125:00 +CPU= 18.929
  08:15:51 STEP 2126 H=2126:00 +CPU= 19.548
  08:16:22 STEP 2127 H=2127:00 +CPU= 30.581
  08:16:42 STEP 2128 H=2128:00 +CPU= 19.508
  08:17:00 STEP 2129 H=2129:00 +CPU= 18.888
  08:17:32 STEP 2130 H=2130:00 +CPU= 31.237
  08:17:51 STEP 2131 H=2131:00 +CPU= 18.882
  08:18:10 STEP 2132 H=2132:00 +CPU= 19.482
  08:18:41 STEP 2133 H=2133:00 +CPU= 30.546
  08:19:00 STEP 2134 H=2134:00 +CPU= 19.475
  08:19:20 STEP 2135 H=2135:00 +CPU= 19.408
  08:19:55 STEP 2136 H=2136:00 +CPU= 35.540
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_88.zip
Uploading the intermediate file: upload_file_88.zip
  08:20:14 STEP 2137 H=2137:00 +CPU= 18.952
  08:20:34 STEP 2138 H=2138:00 +CPU= 19.522
  08:21:04 STEP 2139 H=2139:00 +CPU= 30.588
  08:21:24 STEP 2140 H=2140:00 +CPU= 19.534
  08:21:43 STEP 2141 H=2141:00 +CPU= 18.937
  08:22:14 STEP 2142 H=2142:00 +CPU= 31.357
  08:22:33 STEP 2143 H=2143:00 +CPU= 18.947
  08:22:53 STEP 2144 H=2144:00 +CPU= 19.574
  08:23:23 STEP 2145 H=2145:00 +CPU= 30.614
  08:23:43 STEP 2146 H=2146:00 +CPU= 19.590
  08:24:02 STEP 2147 H=2147:00 +CPU= 18.961
  08:24:36 STEP 2148 H=2148:00 +CPU= 33.848
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002148
  08:24:55 STEP 2149 H=2149:00 +CPU= 18.955
  08:25:14 STEP 2150 H=2150:00 +CPU= 19.569
  08:25:45 STEP 2151 H=2151:00 +CPU= 30.614
  08:26:04 STEP 2152 H=2152:00 +CPU= 19.557
  08:26:23 STEP 2153 H=2153:00 +CPU= 18.918
  08:26:54 STEP 2154 H=2154:00 +CPU= 31.290
  08:27:13 STEP 2155 H=2155:00 +CPU= 18.921
  08:27:33 STEP 2156 H=2156:00 +CPU= 19.517
  08:28:03 STEP 2157 H=2157:00 +CPU= 30.559
  08:28:23 STEP 2158 H=2158:00 +CPU= 19.501
  08:28:42 STEP 2159 H=2159:00 +CPU= 19.407
  08:29:18 STEP 2160 H=2160:00 +CPU= 35.508
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_89.zip
Uploading the intermediate file: upload_file_89.zip
  08:29:37 STEP 2161 H=2161:00 +CPU= 18.958
  08:29:56 STEP 2162 H=2162:00 +CPU= 19.525
  08:30:27 STEP 2163 H=2163:00 +CPU= 30.580
  08:30:47 STEP 2164 H=2164:00 +CPU= 19.522
  08:31:05 STEP 2165 H=2165:00 +CPU= 18.913
  08:31:37 STEP 2166 H=2166:00 +CPU= 31.313
  08:31:56 STEP 2167 H=2167:00 +CPU= 18.951
  08:32:15 STEP 2168 H=2168:00 +CPU= 19.560
  08:32:46 STEP 2169 H=2169:00 +CPU= 30.639
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1983050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0348
(argv4) batchid: 952
(argv5) wuid: 12168992
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/0
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_13eae06deffdc7ebc4adef769ea20955 to: /var/lib/boinc/slots/0/oifs_43r3_ps_0348_1983050100_123_952_12168992.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_0348_1983050100_123_952_12168992.zip
ic_ancil_file: ic_ancil_12168992
ifsdata_file: ifsdata_12168992
climate_data_file: clim_data_12168992
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_636b8e4bcf2948bc431d4fe5ee1074dc to: /var/lib/boinc/slots/0/ic_ancil_12168992.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/0/ic_ancil_12168992.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f70785f7c887b5e310055075155bef95 to: /var/lib/boinc/slots/0/ifsdata/ifsdata_12168992.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/0/ifsdata/ifsdata_12168992.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/0/159l_2/clim_data_12168992.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/0/159l_2/clim_data_12168992.zip
Checking for progress XML file: /var/lib/boinc/slots/0/progress_file_12168992.xml
Opened progress file ok : /var/lib/boinc/slots/0/progress_file_12168992.xml
-- Model is restarting --
Adjusting last_iter, 2169, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/0/progress_file_12168992.xml
last_cpu_time: 51970
upload_file_number: 90
last_iter: 2159
last_upload: 7776000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287
The child process has been launched with process id: 1166
Executing the command: /var/lib/boinc/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1166:1166] [20230114:101412:1673716452:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:14:47 STEP 2160 H=2160:00 +CPU= 33.928
  10:15:06 STEP 2161 H=2161:00 +CPU= 18.904
  10:15:26 STEP 2162 H=2162:00 +CPU= 19.520
  10:15:56 STEP 2163 H=2163:00 +CPU= 30.584
  10:16:16 STEP 2164 H=2164:00 +CPU= 19.532
  10:16:35 STEP 2165 H=2165:00 +CPU= 18.924
  10:17:06 STEP 2166 H=2166:00 +CPU= 31.327
  10:17:25 STEP 2167 H=2167:00 +CPU= 18.942
  10:17:45 STEP 2168 H=2168:00 +CPU= 19.560
  10:18:15 STEP 2169 H=2169:00 +CPU= 30.635
  10:18:35 STEP 2170 H=2170:00 +CPU= 19.563
  10:18:54 STEP 2171 H=2171:00 +CPU= 18.941
  10:19:28 STEP 2172 H=2172:00 +CPU= 33.831
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002172
  10:19:47 STEP 2173 H=2173:00 +CPU= 18.943
  10:20:06 STEP 2174 H=2174:00 +CPU= 19.569
  10:20:37 STEP 2175 H=2175:00 +CPU= 30.636
  10:20:56 STEP 2176 H=2176:00 +CPU= 19.542
  10:21:15 STEP 2177 H=2177:00 +CPU= 18.915
  10:21:47 STEP 2178 H=2178:00 +CPU= 31.319
  10:22:05 STEP 2179 H=2179:00 +CPU= 18.899
  10:22:25 STEP 2180 H=2180:00 +CPU= 19.517
  10:22:56 STEP 2181 H=2181:00 +CPU= 30.602
  10:23:15 STEP 2182 H=2182:00 +CPU= 19.506
  10:23:34 STEP 2183 H=2183:00 +CPU= 19.417
  10:24:10 STEP 2184 H=2184:00 +CPU= 35.834
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_90.zip
Uploading the intermediate file: upload_file_90.zip
  10:24:29 STEP 2185 H=2185:00 +CPU= 19.070
  10:24:49 STEP 2186 H=2186:00 +CPU= 19.600
  10:25:20 STEP 2187 H=2187:00 +CPU= 30.668
  10:25:39 STEP 2188 H=2188:00 +CPU= 19.609
  10:25:58 STEP 2189 H=2189:00 +CPU= 18.987
  10:26:30 STEP 2190 H=2190:00 +CPU= 31.392
  10:26:49 STEP 2191 H=2191:00 +CPU= 19.005
  10:27:08 STEP 2192 H=2192:00 +CPU= 19.608
  10:27:39 STEP 2193 H=2193:00 +CPU= 30.687
  10:27:59 STEP 2194 H=2194:00 +CPU= 19.635
  10:28:18 STEP 2195 H=2195:00 +CPU= 18.996
  10:28:51 STEP 2196 H=2196:00 +CPU= 33.899
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002196
  10:29:11 STEP 2197 H=2197:00 +CPU= 19.029
  10:29:30 STEP 2198 H=2198:00 +CPU= 19.796
  10:30:01 STEP 2199 H=2199:00 +CPU= 30.692
  10:30:21 STEP 2200 H=2200:00 +CPU= 19.588
  10:30:40 STEP 2201 H=2201:00 +CPU= 18.963
  10:31:11 STEP 2202 H=2202:00 +CPU= 31.349
  10:31:30 STEP 2203 H=2203:00 +CPU= 18.951
  10:31:49 STEP 2204 H=2204:00 +CPU= 19.563
  10:32:20 STEP 2205 H=2205:00 +CPU= 30.652
  10:32:40 STEP 2206 H=2206:00 +CPU= 19.573
  10:32:59 STEP 2207 H=2207:00 +CPU= 19.488
  10:33:35 STEP 2208 H=2208:00 +CPU= 35.570
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_91.zip
Uploading the intermediate file: upload_file_91.zip
  10:33:54 STEP 2209 H=2209:00 +CPU= 18.995
  10:34:13 STEP 2210 H=2210:00 +CPU= 19.552
  10:34:44 STEP 2211 H=2211:00 +CPU= 30.634
  10:35:04 STEP 2212 H=2212:00 +CPU= 19.565
  10:35:22 STEP 2213 H=2213:00 +CPU= 18.965
  10:35:54 STEP 2214 H=2214:00 +CPU= 31.355
  10:36:13 STEP 2215 H=2215:00 +CPU= 19.006
  10:36:32 STEP 2216 H=2216:00 +CPU= 19.623
  10:37:03 STEP 2217 H=2217:00 +CPU= 30.673
  10:37:23 STEP 2218 H=2218:00 +CPU= 19.617
  10:37:42 STEP 2219 H=2219:00 +CPU= 18.983
  10:38:16 STEP 2220 H=2220:00 +CPU= 33.862
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002220
  10:38:35 STEP 2221 H=2221:00 +CPU= 18.973
  10:38:54 STEP 2222 H=2222:00 +CPU= 19.598
  10:39:25 STEP 2223 H=2223:00 +CPU= 30.625
  10:39:44 STEP 2224 H=2224:00 +CPU= 19.572
  10:40:03 STEP 2225 H=2225:00 +CPU= 18.946
  10:40:35 STEP 2226 H=2226:00 +CPU= 31.335
  10:40:54 STEP 2227 H=2227:00 +CPU= 18.925
  10:41:13 STEP 2228 H=2228:00 +CPU= 19.553
  10:41:44 STEP 2229 H=2229:00 +CPU= 30.618
  10:42:03 STEP 2230 H=2230:00 +CPU= 19.538
  10:42:23 STEP 2231 H=2231:00 +CPU= 19.460
  10:42:58 STEP 2232 H=2232:00 +CPU= 35.632
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_92.zip
Uploading the intermediate file: upload_file_92.zip
  10:43:17 STEP 2233 H=2233:00 +CPU= 18.997
  10:43:37 STEP 2234 H=2234:00 +CPU= 19.570
  10:44:08 STEP 2235 H=2235:00 +CPU= 30.648
  10:44:27 STEP 2236 H=2236:00 +CPU= 19.592
  10:44:46 STEP 2237 H=2237:00 +CPU= 18.988
  10:45:18 STEP 2238 H=2238:00 +CPU= 31.399
  10:45:37 STEP 2239 H=2239:00 +CPU= 19.000
  10:45:56 STEP 2240 H=2240:00 +CPU= 19.623
  10:46:27 STEP 2241 H=2241:00 +CPU= 30.680
  10:46:47 STEP 2242 H=2242:00 +CPU= 19.626
  10:47:06 STEP 2243 H=2243:00 +CPU= 18.994
  10:47:40 STEP 2244 H=2244:00 +CPU= 33.894
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002244
  10:47:59 STEP 2245 H=2245:00 +CPU= 18.999
  10:48:18 STEP 2246 H=2246:00 +CPU= 19.622
  10:48:49 STEP 2247 H=2247:00 +CPU= 30.675
  10:49:08 STEP 2248 H=2248:00 +CPU= 19.604
  10:49:27 STEP 2249 H=2249:00 +CPU= 18.980
  10:49:59 STEP 2250 H=2250:00 +CPU= 31.362
  10:50:18 STEP 2251 H=2251:00 +CPU= 18.967
  10:50:37 STEP 2252 H=2252:00 +CPU= 19.588
  10:51:08 STEP 2253 H=2253:00 +CPU= 30.643
  10:51:28 STEP 2254 H=2254:00 +CPU= 19.584
  10:51:47 STEP 2255 H=2255:00 +CPU= 19.500
  10:52:23 STEP 2256 H=2256:00 +CPU= 35.600
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_93.zip
Uploading the intermediate file: upload_file_93.zip
  10:52:42 STEP 2257 H=2257:00 +CPU= 19.008
  10:53:01 STEP 2258 H=2258:00 +CPU= 19.589
  10:53:32 STEP 2259 H=2259:00 +CPU= 30.667
  10:53:52 STEP 2260 H=2260:00 +CPU= 19.593
  10:54:11 STEP 2261 H=2261:00 +CPU= 18.974
  10:54:42 STEP 2262 H=2262:00 +CPU= 31.383
  10:55:01 STEP 2263 H=2263:00 +CPU= 18.981
  10:55:21 STEP 2264 H=2264:00 +CPU= 19.602
  10:55:51 STEP 2265 H=2265:00 +CPU= 30.658
  10:56:11 STEP 2266 H=2266:00 +CPU= 19.591
  10:56:30 STEP 2267 H=2267:00 +CPU= 18.983
  10:57:04 STEP 2268 H=2268:00 +CPU= 33.877
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002268
  10:57:23 STEP 2269 H=2269:00 +CPU= 19.004
  10:57:42 STEP 2270 H=2270:00 +CPU= 19.619
  10:58:13 STEP 2271 H=2271:00 +CPU= 30.665
  10:58:33 STEP 2272 H=2272:00 +CPU= 19.593
  10:58:51 STEP 2273 H=2273:00 +CPU= 18.959
  10:59:23 STEP 2274 H=2274:00 +CPU= 31.361
  10:59:42 STEP 2275 H=2275:00 +CPU= 18.955
  11:00:01 STEP 2276 H=2276:00 +CPU= 19.557
  11:00:32 STEP 2277 H=2277:00 +CPU= 30.653
  11:00:52 STEP 2278 H=2278:00 +CPU= 19.545
  11:01:11 STEP 2279 H=2279:00 +CPU= 19.467
  11:01:47 STEP 2280 H=2280:00 +CPU= 35.641
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_94.zip
Uploading the intermediate file: upload_file_94.zip
  11:02:06 STEP 2281 H=2281:00 +CPU= 19.007
  11:02:25 STEP 2282 H=2282:00 +CPU= 19.561
  11:02:56 STEP 2283 H=2283:00 +CPU= 30.668
  11:03:16 STEP 2284 H=2284:00 +CPU= 19.572
  11:03:34 STEP 2285 H=2285:00 +CPU= 18.954
  11:04:06 STEP 2286 H=2286:00 +CPU= 31.400
  11:04:25 STEP 2287 H=2287:00 +CPU= 18.978
  11:04:44 STEP 2288 H=2288:00 +CPU= 19.604
  11:05:15 STEP 2289 H=2289:00 +CPU= 30.699
  11:05:35 STEP 2290 H=2290:00 +CPU= 19.614
  11:05:54 STEP 2291 H=2291:00 +CPU= 18.995
  11:06:28 STEP 2292 H=2292:00 +CPU= 33.890
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002292
  11:06:47 STEP 2293 H=2293:00 +CPU= 19.002
  11:07:06 STEP 2294 H=2294:00 +CPU= 19.636
  11:07:37 STEP 2295 H=2295:00 +CPU= 30.706
  11:07:57 STEP 2296 H=2296:00 +CPU= 19.607
  11:08:16 STEP 2297 H=2297:00 +CPU= 18.981
  11:08:47 STEP 2298 H=2298:00 +CPU= 31.383
  11:09:06 STEP 2299 H=2299:00 +CPU= 18.954
  11:09:26 STEP 2300 H=2300:00 +CPU= 19.562
  11:09:56 STEP 2301 H=2301:00 +CPU= 30.670
  11:10:16 STEP 2302 H=2302:00 +CPU= 19.551
  11:10:35 STEP 2303 H=2303:00 +CPU= 19.468
  11:11:11 STEP 2304 H=2304:00 +CPU= 35.658
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_95.zip
Uploading the intermediate file: upload_file_95.zip
  11:11:30 STEP 2305 H=2305:00 +CPU= 18.999
  11:11:50 STEP 2306 H=2306:00 +CPU= 19.563
  11:12:20 STEP 2307 H=2307:00 +CPU= 30.654
  11:12:40 STEP 2308 H=2308:00 +CPU= 19.552
  11:12:59 STEP 2309 H=2309:00 +CPU= 18.949
  11:13:30 STEP 2310 H=2310:00 +CPU= 31.357
  11:13:49 STEP 2311 H=2311:00 +CPU= 18.971
  11:14:09 STEP 2312 H=2312:00 +CPU= 19.589
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1983050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0348
(argv4) batchid: 952
(argv5) wuid: 12168992
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/0
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_13eae06deffdc7ebc4adef769ea20955 to: /var/lib/boinc/slots/0/oifs_43r3_ps_0348_1983050100_123_952_12168992.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_0348_1983050100_123_952_12168992.zip
ic_ancil_file: ic_ancil_12168992
ifsdata_file: ifsdata_12168992
climate_data_file: clim_data_12168992
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_636b8e4bcf2948bc431d4fe5ee1074dc to: /var/lib/boinc/slots/0/ic_ancil_12168992.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/0/ic_ancil_12168992.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f70785f7c887b5e310055075155bef95 to: /var/lib/boinc/slots/0/ifsdata/ifsdata_12168992.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/0/ifsdata/ifsdata_12168992.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/0/159l_2/clim_data_12168992.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/0/159l_2/clim_data_12168992.zip
Checking for progress XML file: /var/lib/boinc/slots/0/progress_file_12168992.xml
Opened progress file ok : /var/lib/boinc/slots/0/progress_file_12168992.xml
-- Model is restarting --
Adjusting last_iter, 2313, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/0/progress_file_12168992.xml
last_cpu_time: 55586
upload_file_number: 96
last_iter: 2303
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287
The child process has been launched with process id: 1702
Executing the command: /var/lib/boinc/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:1702:1702] [20230114:125727:1673726247:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:58:04 STEP 2304 H=2304:00 +CPU= 34.846
  12:58:23 STEP 2305 H=2305:00 +CPU= 19.474
  12:58:43 STEP 2306 H=2306:00 +CPU= 20.102
  12:59:15 STEP 2307 H=2307:00 +CPU= 31.471
  12:59:35 STEP 2308 H=2308:00 +CPU= 19.907
  12:59:54 STEP 2309 H=2309:00 +CPU= 18.955
  13:00:25 STEP 2310 H=2310:00 +CPU= 31.299
  13:00:44 STEP 2311 H=2311:00 +CPU= 18.938
  13:01:04 STEP 2312 H=2312:00 +CPU= 19.564
  13:01:34 STEP 2313 H=2313:00 +CPU= 30.598
  13:01:54 STEP 2314 H=2314:00 +CPU= 19.578
  13:02:13 STEP 2315 H=2315:00 +CPU= 18.986
  13:02:47 STEP 2316 H=2316:00 +CPU= 33.782
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002316
  13:03:05 STEP 2317 H=2317:00 +CPU= 18.948
  13:03:25 STEP 2318 H=2318:00 +CPU= 19.568
  13:03:56 STEP 2319 H=2319:00 +CPU= 30.583
  13:04:15 STEP 2320 H=2320:00 +CPU= 19.554
  13:04:34 STEP 2321 H=2321:00 +CPU= 18.931
  13:05:05 STEP 2322 H=2322:00 +CPU= 31.276
  13:05:24 STEP 2323 H=2323:00 +CPU= 18.925
  13:05:44 STEP 2324 H=2324:00 +CPU= 19.548
  13:06:14 STEP 2325 H=2325:00 +CPU= 30.565
  13:06:34 STEP 2326 H=2326:00 +CPU= 19.537
  13:06:53 STEP 2327 H=2327:00 +CPU= 19.444
  13:07:29 STEP 2328 H=2328:00 +CPU= 35.937
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_96.zip
Uploading the intermediate file: upload_file_96.zip
  13:07:49 STEP 2329 H=2329:00 +CPU= 19.127
  13:08:08 STEP 2330 H=2330:00 +CPU= 19.686
  13:08:39 STEP 2331 H=2331:00 +CPU= 30.935
  13:08:59 STEP 2332 H=2332:00 +CPU= 19.703
  13:09:18 STEP 2333 H=2333:00 +CPU= 19.045
  13:09:50 STEP 2334 H=2334:00 +CPU= 31.617
  13:10:09 STEP 2335 H=2335:00 +CPU= 19.079
  13:10:28 STEP 2336 H=2336:00 +CPU= 19.689
  13:10:59 STEP 2337 H=2337:00 +CPU= 30.902
  13:11:19 STEP 2338 H=2338:00 +CPU= 19.695
  13:11:38 STEP 2339 H=2339:00 +CPU= 19.062
  13:12:12 STEP 2340 H=2340:00 +CPU= 34.079
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002340
  13:12:31 STEP 2341 H=2341:00 +CPU= 19.084
  13:12:51 STEP 2342 H=2342:00 +CPU= 19.696
  13:13:22 STEP 2343 H=2343:00 +CPU= 30.872
  13:13:41 STEP 2344 H=2344:00 +CPU= 19.674
  13:14:00 STEP 2345 H=2345:00 +CPU= 19.034
  13:14:32 STEP 2346 H=2346:00 +CPU= 31.593
  13:14:51 STEP 2347 H=2347:00 +CPU= 19.063
  13:15:11 STEP 2348 H=2348:00 +CPU= 19.688
  13:15:42 STEP 2349 H=2349:00 +CPU= 30.869
  13:16:01 STEP 2350 H=2350:00 +CPU= 19.648
  13:16:21 STEP 2351 H=2351:00 +CPU= 19.587
  13:16:57 STEP 2352 H=2352:00 +CPU= 35.577
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_97.zip
Uploading the intermediate file: upload_file_97.zip
  13:17:16 STEP 2353 H=2353:00 +CPU= 19.007
  13:17:35 STEP 2354 H=2354:00 +CPU= 19.585
  13:18:06 STEP 2355 H=2355:00 +CPU= 30.642
  13:18:25 STEP 2356 H=2356:00 +CPU= 19.594
  13:18:44 STEP 2357 H=2357:00 +CPU= 18.963
  13:19:16 STEP 2358 H=2358:00 +CPU= 31.358
  13:19:35 STEP 2359 H=2359:00 +CPU= 18.979
  13:19:54 STEP 2360 H=2360:00 +CPU= 19.605
  13:20:25 STEP 2361 H=2361:00 +CPU= 30.637
  13:20:45 STEP 2362 H=2362:00 +CPU= 19.596
  13:21:03 STEP 2363 H=2363:00 +CPU= 18.972
  13:21:37 STEP 2364 H=2364:00 +CPU= 33.792
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002364
  13:21:56 STEP 2365 H=2365:00 +CPU= 18.966
  13:22:16 STEP 2366 H=2366:00 +CPU= 19.589
  13:22:46 STEP 2367 H=2367:00 +CPU= 30.592
  13:23:06 STEP 2368 H=2368:00 +CPU= 19.576
  13:23:25 STEP 2369 H=2369:00 +CPU= 18.954
  13:23:56 STEP 2370 H=2370:00 +CPU= 31.294
  13:24:15 STEP 2371 H=2371:00 +CPU= 18.974
  13:24:35 STEP 2372 H=2372:00 +CPU= 19.573
  13:25:05 STEP 2373 H=2373:00 +CPU= 30.619
  13:25:25 STEP 2374 H=2374:00 +CPU= 19.562
  13:25:45 STEP 2375 H=2375:00 +CPU= 19.506
  13:26:20 STEP 2376 H=2376:00 +CPU= 35.805
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_98.zip
Uploading the intermediate file: upload_file_98.zip
  13:26:39 STEP 2377 H=2377:00 +CPU= 19.068
  13:26:59 STEP 2378 H=2378:00 +CPU= 19.608
  13:27:30 STEP 2379 H=2379:00 +CPU= 30.744
  13:27:49 STEP 2380 H=2380:00 +CPU= 19.629
  13:28:08 STEP 2381 H=2381:00 +CPU= 19.024
  13:28:40 STEP 2382 H=2382:00 +CPU= 31.468
  13:28:59 STEP 2383 H=2383:00 +CPU= 19.034
  13:29:19 STEP 2384 H=2384:00 +CPU= 19.634
  13:29:49 STEP 2385 H=2385:00 +CPU= 30.773
  13:30:09 STEP 2386 H=2386:00 +CPU= 19.646
  13:30:28 STEP 2387 H=2387:00 +CPU= 19.012
  13:31:02 STEP 2388 H=2388:00 +CPU= 33.957
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002388
  13:31:21 STEP 2389 H=2389:00 +CPU= 19.008
  13:31:41 STEP 2390 H=2390:00 +CPU= 19.623
  13:32:11 STEP 2391 H=2391:00 +CPU= 30.710
  13:32:31 STEP 2392 H=2392:00 +CPU= 19.608
  13:32:50 STEP 2393 H=2393:00 +CPU= 18.988
  13:33:21 STEP 2394 H=2394:00 +CPU= 31.423
  13:33:40 STEP 2395 H=2395:00 +CPU= 18.995
  13:34:00 STEP 2396 H=2396:00 +CPU= 19.588
  13:34:31 STEP 2397 H=2397:00 +CPU= 30.703
  13:34:50 STEP 2398 H=2398:00 +CPU= 19.573
  13:35:10 STEP 2399 H=2399:00 +CPU= 19.502
  13:35:45 STEP 2400 H=2400:00 +CPU= 35.623
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_99.zip
Uploading the intermediate file: upload_file_99.zip
  13:36:04 STEP 2401 H=2401:00 +CPU= 18.967
  13:36:24 STEP 2402 H=2402:00 +CPU= 19.580
  13:36:55 STEP 2403 H=2403:00 +CPU= 30.626
  13:37:14 STEP 2404 H=2404:00 +CPU= 19.578
  13:37:33 STEP 2405 H=2405:00 +CPU= 18.972
  13:38:05 STEP 2406 H=2406:00 +CPU= 31.333
  13:38:24 STEP 2407 H=2407:00 +CPU= 19.000
  13:38:43 STEP 2408 H=2408:00 +CPU= 19.586
  13:39:14 STEP 2409 H=2409:00 +CPU= 30.666
  13:39:33 STEP 2410 H=2410:00 +CPU= 19.606
  13:39:52 STEP 2411 H=2411:00 +CPU= 18.977
  13:40:26 STEP 2412 H=2412:00 +CPU= 33.849
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002412
  13:40:45 STEP 2413 H=2413:00 +CPU= 18.980
  13:41:05 STEP 2414 H=2414:00 +CPU= 19.604
  13:41:36 STEP 2415 H=2415:00 +CPU= 30.820
  13:41:55 STEP 2416 H=2416:00 +CPU= 19.696
  13:42:14 STEP 2417 H=2417:00 +CPU= 19.088
  13:42:46 STEP 2418 H=2418:00 +CPU= 31.518
  13:43:05 STEP 2419 H=2419:00 +CPU= 19.069
  13:43:25 STEP 2420 H=2420:00 +CPU= 19.675
  13:43:56 STEP 2421 H=2421:00 +CPU= 30.828
  13:44:15 STEP 2422 H=2422:00 +CPU= 19.678
  13:44:35 STEP 2423 H=2423:00 +CPU= 19.604
  13:45:11 STEP 2424 H=2424:00 +CPU= 35.979
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_100.zip
Uploading the intermediate file: upload_file_100.zip
  13:45:30 STEP 2425 H=2425:00 +CPU= 19.042
  13:45:49 STEP 2426 H=2426:00 +CPU= 19.604
  13:46:20 STEP 2427 H=2427:00 +CPU= 30.714
  13:46:40 STEP 2428 H=2428:00 +CPU= 19.740
  13:46:59 STEP 2429 H=2429:00 +CPU= 19.038
  13:47:30 STEP 2430 H=2430:00 +CPU= 31.433
  13:47:49 STEP 2431 H=2431:00 +CPU= 19.051
  13:48:09 STEP 2432 H=2432:00 +CPU= 19.602
  13:48:40 STEP 2433 H=2433:00 +CPU= 30.644
  13:48:59 STEP 2434 H=2434:00 +CPU= 19.603
  13:49:18 STEP 2435 H=2435:00 +CPU= 18.992
  13:49:52 STEP 2436 H=2436:00 +CPU= 33.856
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002436
  13:50:11 STEP 2437 H=2437:00 +CPU= 18.985
  13:50:31 STEP 2438 H=2438:00 +CPU= 19.604
  13:51:01 STEP 2439 H=2439:00 +CPU= 30.656
  13:51:21 STEP 2440 H=2440:00 +CPU= 19.557
  13:51:40 STEP 2441 H=2441:00 +CPU= 18.931
  13:52:11 STEP 2442 H=2442:00 +CPU= 31.284
  13:52:30 STEP 2443 H=2443:00 +CPU= 18.915
  13:52:50 STEP 2444 H=2444:00 +CPU= 19.529
  13:53:20 STEP 2445 H=2445:00 +CPU= 30.594
  13:53:40 STEP 2446 H=2446:00 +CPU= 19.554
  13:53:59 STEP 2447 H=2447:00 +CPU= 19.458
  13:54:35 STEP 2448 H=2448:00 +CPU= 35.588
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_101.zip
Uploading the intermediate file: upload_file_101.zip
  13:54:54 STEP 2449 H=2449:00 +CPU= 18.999
  13:55:13 STEP 2450 H=2450:00 +CPU= 19.546
  13:55:44 STEP 2451 H=2451:00 +CPU= 30.857
  13:56:04 STEP 2452 H=2452:00 +CPU= 19.614
  13:56:23 STEP 2453 H=2453:00 +CPU= 18.956
  13:56:54 STEP 2454 H=2454:00 +CPU= 31.341
  13:57:13 STEP 2455 H=2455:00 +CPU= 18.961
  13:57:33 STEP 2456 H=2456:00 +CPU= 19.582
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1983050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0348
(argv4) batchid: 952
(argv5) wuid: 12168992
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/0
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_13eae06deffdc7ebc4adef769ea20955 to: /var/lib/boinc/slots/0/oifs_43r3_ps_0348_1983050100_123_952_12168992.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_0348_1983050100_123_952_12168992.zip
ic_ancil_file: ic_ancil_12168992
ifsdata_file: ifsdata_12168992
climate_data_file: clim_data_12168992
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_636b8e4bcf2948bc431d4fe5ee1074dc to: /var/lib/boinc/slots/0/ic_ancil_12168992.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/0/ic_ancil_12168992.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f70785f7c887b5e310055075155bef95 to: /var/lib/boinc/slots/0/ifsdata/ifsdata_12168992.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/0/ifsdata/ifsdata_12168992.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/0/159l_2/clim_data_12168992.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/0/159l_2/clim_data_12168992.zip
Checking for progress XML file: /var/lib/boinc/slots/0/progress_file_12168992.xml
Opened progress file ok : /var/lib/boinc/slots/0/progress_file_12168992.xml
-- Model is restarting --
Adjusting last_iter, 2457, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/0/progress_file_12168992.xml
last_cpu_time: 59202
upload_file_number: 102
last_iter: 2447
last_upload: 8812800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287
The child process has been launched with process id: 2331
Executing the command: /var/lib/boinc/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:2331:2331] [20230114:155638:1673736998:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:57:14 STEP 2448 H=2448:00 +CPU= 33.902
  15:57:33 STEP 2449 H=2449:00 +CPU= 18.914
  15:57:52 STEP 2450 H=2450:00 +CPU= 19.506
  15:58:23 STEP 2451 H=2451:00 +CPU= 30.582
  15:58:42 STEP 2452 H=2452:00 +CPU= 19.506
  15:59:01 STEP 2453 H=2453:00 +CPU= 18.909
  15:59:33 STEP 2454 H=2454:00 +CPU= 31.300
  15:59:52 STEP 2455 H=2455:00 +CPU= 18.937
  16:00:11 STEP 2456 H=2456:00 +CPU= 19.563
  16:00:42 STEP 2457 H=2457:00 +CPU= 30.631
  16:01:01 STEP 2458 H=2458:00 +CPU= 19.559
  16:01:20 STEP 2459 H=2459:00 +CPU= 18.935
  16:01:54 STEP 2460 H=2460:00 +CPU= 33.789
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002460
  16:02:13 STEP 2461 H=2461:00 +CPU= 18.928
  16:02:33 STEP 2462 H=2462:00 +CPU= 19.544
  16:03:03 STEP 2463 H=2463:00 +CPU= 30.600
  16:03:23 STEP 2464 H=2464:00 +CPU= 19.525
  16:03:42 STEP 2465 H=2465:00 +CPU= 18.910
  16:04:13 STEP 2466 H=2466:00 +CPU= 31.284
  16:04:32 STEP 2467 H=2467:00 +CPU= 18.904
  16:04:51 STEP 2468 H=2468:00 +CPU= 19.531
  16:05:22 STEP 2469 H=2469:00 +CPU= 30.560
  16:05:41 STEP 2470 H=2470:00 +CPU= 19.508
  16:06:01 STEP 2471 H=2471:00 +CPU= 19.413
  16:06:37 STEP 2472 H=2472:00 +CPU= 35.897
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12168992/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0348_1983050100_123_952_12168992_0_r637163287_102.zip
Uploading the intermediate file: upload_file_102.zip
  16:06:56 STEP 2473 H=2473:00 +CPU= 19.068
  16:07:15 STEP 2474 H=2474:00 +CPU= 19.627
  16:07:46 STEP 2475 H=2475:00 +CPU= 30.693
  16:08:06 STEP 2476 H=2476:00 +CPU= 19.620
  16:08:25 STEP 2477 H=2477:00 +CPU= 19.003
  16:08:56 STEP 2478 H=2478:00 +CPU= 31.359
  16:09:15 STEP 2479 H=2479:00 +CPU= 19.020
  16:09:35 STEP 2480 H=2480:00 +CPU= 19.651
  16:10:06 STEP 2481 H=2481:00 +CPU= 30.777
  16:10:25 STEP 2482 H=2482:00 +CPU= 19.763
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:18:42 1463165 22253270 oifs_43r3_ps_0348_1983050100_123_952_12168992_0 4,320,000 28,119 0.0065
09 May 2023 16:18:38 1463165 22253270 oifs_43r3_ps_0348_1983050100_123_952_12168992_0 3,456,000 22,506 0.0065
09 May 2023 16:18:33 1463165 22253270 oifs_43r3_ps_0348_1983050100_123_952_12168992_0 2,592,000 16,874 0.0065
09 May 2023 16:18:28 1463165 22253270 oifs_43r3_ps_0348_1983050100_123_952_12168992_0 1,728,000 11,262 0.0065
09 May 2023 16:18:24 1463165 22253270 oifs_43r3_ps_0348_1983050100_123_952_12168992_0 864,000 5,650 0.0065


©2024 cpdn.org