climateprediction.net home page
Task 22301921

Task 22301921

Name oifs_43r3_ps_0388_2008050100_123_977_12194032_1
Workunit 12194032
Created 22 Jan 2023, 18:32:28 UTC
Sent 22 Jan 2023, 19:02:04 UTC
Report deadline 23 Mar 2023, 19:02:04 UTC
Received 23 Jan 2023, 10:52:32 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1512026
Run time 12 hours 33 min 18 sec
CPU time 11 hours 57 min 20 sec
Validate state Invalid
Credit 11,128.32
Device peak FLOPS 6.68 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,404.02 MB
Peak swap size 4,978.17 MB
Peak disk usage 7,469.43 MB
Stderr
<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_91.zip
Uploading the intermediate file: upload_file_91.zip
  07:58:40 STEP 2209 H=2209:00 +CPU= 11.959
  07:58:52 STEP 2210 H=2210:00 +CPU= 12.236
Uploading trickle at timestep: 7948800
  07:59:18 STEP 2211 H=2211:00 +CPU= 23.127
  07:59:33 STEP 2212 H=2212:00 +CPU= 15.202
  07:59:45 STEP 2213 H=2213:00 +CPU= 11.974
  08:00:07 STEP 2214 H=2214:00 +CPU= 21.975
  08:00:23 STEP 2215 H=2215:00 +CPU= 15.244
  08:00:36 STEP 2216 H=2216:00 +CPU= 12.980
  08:00:58 STEP 2217 H=2217:00 +CPU= 22.072
  08:01:12 STEP 2218 H=2218:00 +CPU= 13.914
  08:01:24 STEP 2219 H=2219:00 +CPU= 12.374
  08:01:46 STEP 2220 H=2220:00 +CPU= 21.632
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002220
  08:02:03 STEP 2221 H=2221:00 +CPU= 14.139
  08:02:19 STEP 2222 H=2222:00 +CPU= 15.914
  08:02:43 STEP 2223 H=2223:00 +CPU= 24.136
  08:02:59 STEP 2224 H=2224:00 +CPU= 15.939
  08:03:12 STEP 2225 H=2225:00 +CPU= 12.573
  08:03:35 STEP 2226 H=2226:00 +CPU= 22.914
  08:03:50 STEP 2227 H=2227:00 +CPU= 15.232
  08:04:04 STEP 2228 H=2228:00 +CPU= 13.287
  08:04:23 STEP 2229 H=2229:00 +CPU= 19.035
  08:04:39 STEP 2230 H=2230:00 +CPU= 15.711
  08:04:52 STEP 2231 H=2231:00 +CPU= 12.661
  08:05:24 STEP 2232 H=2232:00 +CPU= 23.978
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_92.zip
Uploading the intermediate file: upload_file_92.zip
  08:05:36 STEP 2233 H=2233:00 +CPU= 12.239
  08:05:49 STEP 2234 H=2234:00 +CPU= 12.567
  08:06:08 STEP 2235 H=2235:00 +CPU= 19.118
  08:06:20 STEP 2236 H=2236:00 +CPU= 12.637
  08:06:33 STEP 2237 H=2237:00 +CPU= 12.168
  08:06:52 STEP 2238 H=2238:00 +CPU= 19.300
  08:07:05 STEP 2239 H=2239:00 +CPU= 12.822
  08:07:19 STEP 2240 H=2240:00 +CPU= 13.647
  08:07:38 STEP 2241 H=2241:00 +CPU= 18.953
  08:07:52 STEP 2242 H=2242:00 +CPU= 14.280
  08:08:06 STEP 2243 H=2243:00 +CPU= 14.156
  08:08:27 STEP 2244 H=2244:00 +CPU= 21.123
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002244
  08:08:40 STEP 2245 H=2245:00 +CPU= 12.149
  08:08:52 STEP 2246 H=2246:00 +CPU= 12.688
  08:09:12 STEP 2247 H=2247:00 +CPU= 19.638
  08:09:25 STEP 2248 H=2248:00 +CPU= 12.667
  08:09:37 STEP 2249 H=2249:00 +CPU= 12.292
  08:09:57 STEP 2250 H=2250:00 +CPU= 19.887
  08:10:09 STEP 2251 H=2251:00 +CPU= 12.317
  08:10:22 STEP 2252 H=2252:00 +CPU= 12.339
  08:10:41 STEP 2253 H=2253:00 +CPU= 19.629
  08:10:54 STEP 2254 H=2254:00 +CPU= 12.154
  08:11:06 STEP 2255 H=2255:00 +CPU= 12.390
  08:11:34 STEP 2256 H=2256:00 +CPU= 22.507
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_93.zip
Uploading the intermediate file: upload_file_93.zip
  08:11:46 STEP 2257 H=2257:00 +CPU= 12.017
  08:11:58 STEP 2258 H=2258:00 +CPU= 12.151
  08:12:20 STEP 2259 H=2259:00 +CPU= 21.569
  08:12:32 STEP 2260 H=2260:00 +CPU= 12.625
  08:12:45 STEP 2261 H=2261:00 +CPU= 12.220
  08:13:04 STEP 2262 H=2262:00 +CPU= 19.512
  08:13:19 STEP 2263 H=2263:00 +CPU= 13.226
  08:13:32 STEP 2264 H=2264:00 +CPU= 12.305
  08:13:51 STEP 2265 H=2265:00 +CPU= 18.978
  08:14:03 STEP 2266 H=2266:00 +CPU= 12.093
  08:14:15 STEP 2267 H=2267:00 +CPU= 11.793
  08:14:36 STEP 2268 H=2268:00 +CPU= 21.111
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002268
  08:14:48 STEP 2269 H=2269:00 +CPU= 11.950
  08:15:01 STEP 2270 H=2270:00 +CPU= 12.671
  08:15:20 STEP 2271 H=2271:00 +CPU= 19.804
  08:15:33 STEP 2272 H=2272:00 +CPU= 12.646
  08:15:45 STEP 2273 H=2273:00 +CPU= 12.267
  08:16:07 STEP 2274 H=2274:00 +CPU= 21.598
  08:16:20 STEP 2275 H=2275:00 +CPU= 12.630
  08:16:34 STEP 2276 H=2276:00 +CPU= 14.273
  08:16:56 STEP 2277 H=2277:00 +CPU= 21.530
  08:17:10 STEP 2278 H=2278:00 +CPU= 14.081
  08:17:24 STEP 2279 H=2279:00 +CPU= 14.202
  08:17:56 STEP 2280 H=2280:00 +CPU= 29.823
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_94.zip
Uploading the intermediate file: upload_file_94.zip
  08:18:12 STEP 2281 H=2281:00 +CPU= 16.066
  08:18:27 STEP 2282 H=2282:00 +CPU= 14.659
  08:18:48 STEP 2283 H=2283:00 +CPU= 21.634
  08:19:04 STEP 2284 H=2284:00 +CPU= 14.996
  08:19:17 STEP 2285 H=2285:00 +CPU= 13.467
  08:19:37 STEP 2286 H=2286:00 +CPU= 20.050
  08:19:53 STEP 2287 H=2287:00 +CPU= 15.911
  08:20:10 STEP 2288 H=2288:00 +CPU= 16.535
  08:20:32 STEP 2289 H=2289:00 +CPU= 22.486
  08:20:46 STEP 2290 H=2290:00 +CPU= 13.992
  08:21:00 STEP 2291 H=2291:00 +CPU= 13.994
  08:21:25 STEP 2292 H=2292:00 +CPU= 23.929
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002292
  08:21:37 STEP 2293 H=2293:00 +CPU= 12.850
  08:21:51 STEP 2294 H=2294:00 +CPU= 13.972
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0388
(argv4) batchid: 977
(argv5) wuid: 12194032
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_272688b46a588a035e513e3503305e34 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0388_2008050100_123_977_12194032.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0388_2008050100_123_977_12194032.zip
ic_ancil_file: ic_ancil_12194032
ifsdata_file: ifsdata_12194032
climate_data_file: clim_data_12194032
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2c93197ab0033a7ad901f882d7d53740 to: /var/lib/boinc-client/slots/5/ic_ancil_12194032.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12194032.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e83bbfe4865fac1c0d2e2ca05beaac19 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12194032.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12194032.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12194032.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12194032.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12194032.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12194032.xml
-- Model is restarting --
Adjusting last_iter, 2295, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12194032.xml
last_cpu_time: 36800
upload_file_number: 95
last_iter: 2279
last_upload: 8208000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284
The child process has been launched with process id: 18133
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18133:18133] [20230123:082359:1674458639:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:24:55 STEP 2280 H=2280:00 +CPU= 27.734
  08:25:10 STEP 2281 H=2281:00 +CPU= 12.244
  08:25:24 STEP 2282 H=2282:00 +CPU= 12.271
  08:25:43 STEP 2283 H=2283:00 +CPU= 18.789
  08:25:55 STEP 2284 H=2284:00 +CPU= 11.999
  08:26:07 STEP 2285 H=2285:00 +CPU= 11.690
  08:26:26 STEP 2286 H=2286:00 +CPU= 19.147
  08:26:38 STEP 2287 H=2287:00 +CPU= 11.790
  08:26:50 STEP 2288 H=2288:00 +CPU= 12.448
  08:27:10 STEP 2289 H=2289:00 +CPU= 19.421
  08:27:25 STEP 2290 H=2290:00 +CPU= 13.217
  08:27:37 STEP 2291 H=2291:00 +CPU= 12.132
  08:28:00 STEP 2292 H=2292:00 +CPU= 21.644
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002292
  08:28:13 STEP 2293 H=2293:00 +CPU= 11.998
  08:28:26 STEP 2294 H=2294:00 +CPU= 12.274
  08:28:45 STEP 2295 H=2295:00 +CPU= 19.350
  08:28:58 STEP 2296 H=2296:00 +CPU= 12.922
  08:29:11 STEP 2297 H=2297:00 +CPU= 12.078
  08:29:37 STEP 2298 H=2298:00 +CPU= 23.227
  08:29:53 STEP 2299 H=2299:00 +CPU= 14.489
  08:30:06 STEP 2300 H=2300:00 +CPU= 12.586
  08:30:27 STEP 2301 H=2301:00 +CPU= 19.151
  08:30:40 STEP 2302 H=2302:00 +CPU= 12.293
  08:30:52 STEP 2303 H=2303:00 +CPU= 12.198
  08:31:34 STEP 2304 H=2304:00 +CPU= 30.996
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_95.zip
Uploading the intermediate file: upload_file_95.zip
  08:31:50 STEP 2305 H=2305:00 +CPU= 14.786
  08:32:04 STEP 2306 H=2306:00 +CPU= 13.484
  08:32:23 STEP 2307 H=2307:00 +CPU= 19.295
  08:32:36 STEP 2308 H=2308:00 +CPU= 12.449
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           29  MAX ETADOT VERTICAL VEL.=   7.544493995745713E-008
  LON  =    12.6562500000000       degrees
  LAT  =   -43.1773092288349       degrees
  08:32:50 STEP 2309 H=2309:00 +CPU= 13.912
  08:33:15 STEP 2310 H=2310:00 +CPU= 24.992
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           30  MAX ETADOT VERTICAL VEL.=   7.386233788924770E-008
  LON  =    14.0625000000000       degrees
  LAT  =   -44.2987941306940       degrees
  08:33:29 STEP 2311 H=2311:00 +CPU= 13.783
  08:33:42 STEP 2312 H=2312:00 +CPU= 12.591
  08:34:03 STEP 2313 H=2313:00 +CPU= 20.417
  08:34:17 STEP 2314 H=2314:00 +CPU= 13.773
  08:34:29 STEP 2315 H=2315:00 +CPU= 11.931
  08:34:52 STEP 2316 H=2316:00 +CPU= 23.244
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002316
  08:35:05 STEP 2317 H=2317:00 +CPU= 12.564
  08:35:20 STEP 2318 H=2318:00 +CPU= 14.697
  08:35:39 STEP 2319 H=2319:00 +CPU= 19.533
  08:36:02 STEP 2320 H=2320:00 +CPU= 15.064
  08:36:14 STEP 2321 H=2321:00 +CPU= 12.280
  08:36:39 STEP 2322 H=2322:00 +CPU= 24.221
  08:36:53 STEP 2323 H=2323:00 +CPU= 13.805
  08:37:10 STEP 2324 H=2324:00 +CPU= 14.059
  08:37:31 STEP 2325 H=2325:00 +CPU= 20.533
  08:37:47 STEP 2326 H=2326:00 +CPU= 15.212
  08:38:01 STEP 2327 H=2327:00 +CPU= 14.414
  08:38:30 STEP 2328 H=2328:00 +CPU= 23.507
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_96.zip
Uploading the intermediate file: upload_file_96.zip
  08:38:42 STEP 2329 H=2329:00 +CPU= 12.247
  08:38:55 STEP 2330 H=2330:00 +CPU= 12.403
  08:39:17 STEP 2331 H=2331:00 +CPU= 22.184
  08:39:33 STEP 2332 H=2332:00 +CPU= 15.227
  08:39:48 STEP 2333 H=2333:00 +CPU= 15.080
  08:40:12 STEP 2334 H=2334:00 +CPU= 23.346
  08:40:29 STEP 2335 H=2335:00 +CPU= 15.031
  08:40:41 STEP 2336 H=2336:00 +CPU= 12.293
  08:41:01 STEP 2337 H=2337:00 +CPU= 19.036
  08:41:13 STEP 2338 H=2338:00 +CPU= 12.664
  08:41:27 STEP 2339 H=2339:00 +CPU= 13.635
  08:41:57 STEP 2340 H=2340:00 +CPU= 26.636
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002340
  08:42:11 STEP 2341 H=2341:00 +CPU= 13.853
  08:42:24 STEP 2342 H=2342:00 +CPU= 12.622
  08:42:44 STEP 2343 H=2343:00 +CPU= 19.465
  08:42:56 STEP 2344 H=2344:00 +CPU= 12.163
  08:43:08 STEP 2345 H=2345:00 +CPU= 12.067
  08:43:27 STEP 2346 H=2346:00 +CPU= 19.403
  08:43:40 STEP 2347 H=2347:00 +CPU= 11.948
  08:43:54 STEP 2348 H=2348:00 +CPU= 14.518
  08:44:17 STEP 2349 H=2349:00 +CPU= 23.193
  08:44:30 STEP 2350 H=2350:00 +CPU= 12.776
  08:44:43 STEP 2351 H=2351:00 +CPU= 12.222
  08:45:12 STEP 2352 H=2352:00 +CPU= 23.108
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_97.zip
Uploading the intermediate file: upload_file_97.zip
  08:45:24 STEP 2353 H=2353:00 +CPU= 11.919
  08:45:37 STEP 2354 H=2354:00 +CPU= 12.271
  08:46:00 STEP 2355 H=2355:00 +CPU= 20.706
  08:46:20 STEP 2356 H=2356:00 +CPU= 16.750
  08:46:35 STEP 2357 H=2357:00 +CPU= 15.472
  08:46:57 STEP 2358 H=2358:00 +CPU= 21.680
  08:47:10 STEP 2359 H=2359:00 +CPU= 12.861
  08:47:23 STEP 2360 H=2360:00 +CPU= 12.443
  08:47:47 STEP 2361 H=2361:00 +CPU= 21.917
  08:48:04 STEP 2362 H=2362:00 +CPU= 16.765
  08:48:20 STEP 2363 H=2363:00 +CPU= 16.474
  08:48:48 STEP 2364 H=2364:00 +CPU= 26.887
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002364
  08:49:02 STEP 2365 H=2365:00 +CPU= 13.910
  08:49:15 STEP 2366 H=2366:00 +CPU= 12.568
  08:49:34 STEP 2367 H=2367:00 +CPU= 19.117
  08:49:47 STEP 2368 H=2368:00 +CPU= 13.201
  08:50:02 STEP 2369 H=2369:00 +CPU= 13.962
  08:50:24 STEP 2370 H=2370:00 +CPU= 22.503
  08:50:40 STEP 2371 H=2371:00 +CPU= 15.449
  08:50:53 STEP 2372 H=2372:00 +CPU= 12.972
  08:51:19 STEP 2373 H=2373:00 +CPU= 23.658
  08:51:33 STEP 2374 H=2374:00 +CPU= 14.049
  08:51:46 STEP 2375 H=2375:00 +CPU= 12.274
  08:52:21 STEP 2376 H=2376:00 +CPU= 28.481
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_98.zip
Uploading the intermediate file: upload_file_98.zip
  08:52:34 STEP 2377 H=2377:00 +CPU= 12.864
  08:52:48 STEP 2378 H=2378:00 +CPU= 14.395
  08:53:07 STEP 2379 H=2379:00 +CPU= 18.963
  08:53:20 STEP 2380 H=2380:00 +CPU= 12.453
  08:53:32 STEP 2381 H=2381:00 +CPU= 11.884
  08:53:51 STEP 2382 H=2382:00 +CPU= 19.280
  08:54:03 STEP 2383 H=2383:00 +CPU= 11.994
  08:54:15 STEP 2384 H=2384:00 +CPU= 12.203
  08:54:34 STEP 2385 H=2385:00 +CPU= 19.070
  08:54:47 STEP 2386 H=2386:00 +CPU= 12.220
  08:54:59 STEP 2387 H=2387:00 +CPU= 11.908
  08:55:20 STEP 2388 H=2388:00 +CPU= 21.251
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002388
  08:55:33 STEP 2389 H=2389:00 +CPU= 12.601
  08:55:47 STEP 2390 H=2390:00 +CPU= 14.016
  08:56:06 STEP 2391 H=2391:00 +CPU= 19.183
  08:56:19 STEP 2392 H=2392:00 +CPU= 12.186
  08:56:31 STEP 2393 H=2393:00 +CPU= 11.856
  08:56:50 STEP 2394 H=2394:00 +CPU= 19.249
  08:57:02 STEP 2395 H=2395:00 +CPU= 12.038
  08:57:15 STEP 2396 H=2396:00 +CPU= 12.185
  08:57:34 STEP 2397 H=2397:00 +CPU= 19.272
  08:57:46 STEP 2398 H=2398:00 +CPU= 12.268
  08:57:58 STEP 2399 H=2399:00 +CPU= 12.062
  08:58:30 STEP 2400 H=2400:00 +CPU= 24.963
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_99.zip
Uploading the intermediate file: upload_file_99.zip
  08:58:49 STEP 2401 H=2401:00 +CPU= 16.432
  08:59:10 STEP 2402 H=2402:00 +CPU= 15.794
  08:59:33 STEP 2403 H=2403:00 +CPU= 22.739
  08:59:51 STEP 2404 H=2404:00 +CPU= 13.691
  09:00:05 STEP 2405 H=2405:00 +CPU= 12.660
  09:00:37 STEP 2406 H=2406:00 +CPU= 24.384
  09:00:50 STEP 2407 H=2407:00 +CPU= 13.178
  09:01:09 STEP 2408 H=2408:00 +CPU= 14.575
  09:01:30 STEP 2409 H=2409:00 +CPU= 20.301
  09:01:42 STEP 2410 H=2410:00 +CPU= 12.354
  09:01:54 STEP 2411 H=2411:00 +CPU= 11.818
  09:02:16 STEP 2412 H=2412:00 +CPU= 21.270
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002412
  09:02:33 STEP 2413 H=2413:00 +CPU= 15.963
  09:02:53 STEP 2414 H=2414:00 +CPU= 13.853
  09:03:13 STEP 2415 H=2415:00 +CPU= 19.651
  09:03:26 STEP 2416 H=2416:00 +CPU= 12.943
  09:03:39 STEP 2417 H=2417:00 +CPU= 12.278
  09:03:58 STEP 2418 H=2418:00 +CPU= 19.342
  09:04:11 STEP 2419 H=2419:00 +CPU= 13.006
  09:04:24 STEP 2420 H=2420:00 +CPU= 13.060
  09:04:44 STEP 2421 H=2421:00 +CPU= 19.458
  09:04:57 STEP 2422 H=2422:00 +CPU= 12.609
  09:05:11 STEP 2423 H=2423:00 +CPU= 14.280
  09:05:56 STEP 2424 H=2424:00 +CPU= 25.745
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_100.zip
Uploading the intermediate file: upload_file_100.zip
  09:06:08 STEP 2425 H=2425:00 +CPU= 11.844
  09:06:20 STEP 2426 H=2426:00 +CPU= 12.129
  09:06:39 STEP 2427 H=2427:00 +CPU= 18.823
  09:06:51 STEP 2428 H=2428:00 +CPU= 12.373
  09:07:05 STEP 2429 H=2429:00 +CPU= 13.280
  09:10:23 STEP 2430 H=2430:00 +CPU= 24.899
  09:10:41 STEP 2431 H=2431:00 +CPU= 14.310
  09:11:00 STEP 2432 H=2432:00 +CPU= 15.310
  09:11:23 STEP 2433 H=2433:00 +CPU= 21.224
  09:11:42 STEP 2434 H=2434:00 +CPU= 16.586
  09:11:58 STEP 2435 H=2435:00 +CPU= 16.124
  09:12:26 STEP 2436 H=2436:00 +CPU= 26.078
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002436
  09:12:44 STEP 2437 H=2437:00 +CPU= 15.424
  09:13:28 STEP 2438 H=2438:00 +CPU= 17.701
  09:13:47 STEP 2439 H=2439:00 +CPU= 18.956
  09:13:59 STEP 2440 H=2440:00 +CPU= 12.117
  09:14:11 STEP 2441 H=2441:00 +CPU= 12.018
  09:14:33 STEP 2442 H=2442:00 +CPU= 22.259
  09:14:53 STEP 2443 H=2443:00 +CPU= 14.701
  09:15:07 STEP 2444 H=2444:00 +CPU= 13.120
  09:15:27 STEP 2445 H=2445:00 +CPU= 20.427
  09:15:40 STEP 2446 H=2446:00 +CPU= 13.118
  09:15:53 STEP 2447 H=2447:00 +CPU= 12.408
  09:16:33 STEP 2448 H=2448:00 +CPU= 24.852
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_101.zip
Uploading the intermediate file: upload_file_101.zip
  09:16:45 STEP 2449 H=2449:00 +CPU= 12.179
  09:16:58 STEP 2450 H=2450:00 +CPU= 12.241
  09:17:17 STEP 2451 H=2451:00 +CPU= 18.982
  09:17:30 STEP 2452 H=2452:00 +CPU= 12.751
  09:17:42 STEP 2453 H=2453:00 +CPU= 12.356
  09:18:03 STEP 2454 H=2454:00 +CPU= 20.505
  09:18:15 STEP 2455 H=2455:00 +CPU= 12.074
  09:18:28 STEP 2456 H=2456:00 +CPU= 13.181
  09:18:47 STEP 2457 H=2457:00 +CPU= 19.353
  09:19:00 STEP 2458 H=2458:00 +CPU= 12.423
  09:19:12 STEP 2459 H=2459:00 +CPU= 12.121
  09:19:33 STEP 2460 H=2460:00 +CPU= 21.068
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002460
  09:19:45 STEP 2461 H=2461:00 +CPU= 12.257
  09:20:00 STEP 2462 H=2462:00 +CPU= 12.577
  09:20:21 STEP 2463 H=2463:00 +CPU= 20.531
  09:20:34 STEP 2464 H=2464:00 +CPU= 12.756
  09:20:46 STEP 2465 H=2465:00 +CPU= 12.118
  09:21:06 STEP 2466 H=2466:00 +CPU= 19.771
  09:21:18 STEP 2467 H=2467:00 +CPU= 12.058
  09:21:30 STEP 2468 H=2468:00 +CPU= 12.103
  09:21:50 STEP 2469 H=2469:00 +CPU= 19.942
  09:22:02 STEP 2470 H=2470:00 +CPU= 12.302
  09:22:17 STEP 2471 H=2471:00 +CPU= 14.318
  09:22:56 STEP 2472 H=2472:00 +CPU= 27.088
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_102.zip
Uploading the intermediate file: upload_file_102.zip
  09:23:15 STEP 2473 H=2473:00 +CPU= 15.500
  09:23:29 STEP 2474 H=2474:00 +CPU= 13.113
  09:23:48 STEP 2475 H=2475:00 +CPU= 19.577
  09:24:01 STEP 2476 H=2476:00 +CPU= 12.956
  09:24:14 STEP 2477 H=2477:00 +CPU= 12.239
  09:24:34 STEP 2478 H=2478:00 +CPU= 20.131
  09:24:48 STEP 2479 H=2479:00 +CPU= 14.263
  09:25:01 STEP 2480 H=2480:00 +CPU= 12.826
  09:25:21 STEP 2481 H=2481:00 +CPU= 19.694
  09:25:34 STEP 2482 H=2482:00 +CPU= 12.710
  09:25:46 STEP 2483 H=2483:00 +CPU= 12.348
  09:26:11 STEP 2484 H=2484:00 +CPU= 24.750
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002484
  09:26:25 STEP 2485 H=2485:00 +CPU= 13.782
  09:26:41 STEP 2486 H=2486:00 +CPU= 16.389
  09:27:05 STEP 2487 H=2487:00 +CPU= 22.886
  09:27:21 STEP 2488 H=2488:00 +CPU= 16.511
  09:27:34 STEP 2489 H=2489:00 +CPU= 12.641
  09:27:54 STEP 2490 H=2490:00 +CPU= 20.092
  09:28:07 STEP 2491 H=2491:00 +CPU= 12.439
  09:28:19 STEP 2492 H=2492:00 +CPU= 12.572
  09:28:40 STEP 2493 H=2493:00 +CPU= 21.063
  09:28:55 STEP 2494 H=2494:00 +CPU= 14.206
  09:29:10 STEP 2495 H=2495:00 +CPU= 15.686
  09:29:38 STEP 2496 H=2496:00 +CPU= 25.966
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_103.zip
Uploading the intermediate file: upload_file_103.zip
  09:29:52 STEP 2497 H=2497:00 +CPU= 13.994
  09:30:07 STEP 2498 H=2498:00 +CPU= 14.926
  09:30:31 STEP 2499 H=2499:00 +CPU= 23.963
  09:30:45 STEP 2500 H=2500:00 +CPU= 13.746
  09:30:57 STEP 2501 H=2501:00 +CPU= 12.159
  09:31:20 STEP 2502 H=2502:00 +CPU= 22.604
  09:31:34 STEP 2503 H=2503:00 +CPU= 13.784
  09:31:48 STEP 2504 H=2504:00 +CPU= 14.375
  09:32:12 STEP 2505 H=2505:00 +CPU= 23.227
  09:32:27 STEP 2506 H=2506:00 +CPU= 15.432
  09:32:41 STEP 2507 H=2507:00 +CPU= 13.771
  09:33:09 STEP 2508 H=2508:00 +CPU= 27.525
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002508
  09:33:23 STEP 2509 H=2509:00 +CPU= 13.989
  09:33:37 STEP 2510 H=2510:00 +CPU= 13.907
  09:33:59 STEP 2511 H=2511:00 +CPU= 22.147
  09:34:15 STEP 2512 H=2512:00 +CPU= 16.286
  09:34:30 STEP 2513 H=2513:00 +CPU= 15.021
  09:34:51 STEP 2514 H=2514:00 +CPU= 20.591
  09:35:05 STEP 2515 H=2515:00 +CPU= 13.391
  09:35:18 STEP 2516 H=2516:00 +CPU= 13.039
  09:35:44 STEP 2517 H=2517:00 +CPU= 20.791
  09:35:58 STEP 2518 H=2518:00 +CPU= 13.331
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0388
(argv4) batchid: 977
(argv5) wuid: 12194032
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_272688b46a588a035e513e3503305e34 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0388_2008050100_123_977_12194032.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0388_2008050100_123_977_12194032.zip
ic_ancil_file: ic_ancil_12194032
ifsdata_file: ifsdata_12194032
climate_data_file: clim_data_12194032
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2c93197ab0033a7ad901f882d7d53740 to: /var/lib/boinc-client/slots/5/ic_ancil_12194032.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12194032.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e83bbfe4865fac1c0d2e2ca05beaac19 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12194032.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12194032.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12194032.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12194032.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12194032.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12194032.xml
-- Model is restarting --
Adjusting last_iter, 2519, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12194032.xml
last_cpu_time: 40678
upload_file_number: 104
last_iter: 2495
last_upload: 8985600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284
The child process has been launched with process id: 18265
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc:1:1:18265:18265] [20230123:100852:1674464932:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:09:53 STEP 2496 H=2496:00 +CPU= 26.855
  10:10:22 STEP 2497 H=2497:00 +CPU= 14.141
  10:10:35 STEP 2498 H=2498:00 +CPU= 12.094
  10:10:54 STEP 2499 H=2499:00 +CPU= 19.393
  10:11:09 STEP 2500 H=2500:00 +CPU= 12.913
  10:11:22 STEP 2501 H=2501:00 +CPU= 12.094
  10:11:41 STEP 2502 H=2502:00 +CPU= 19.193
  10:11:53 STEP 2503 H=2503:00 +CPU= 11.788
  10:12:05 STEP 2504 H=2504:00 +CPU= 12.184
  10:12:25 STEP 2505 H=2505:00 +CPU= 19.010
  10:12:38 STEP 2506 H=2506:00 +CPU= 12.685
  10:12:56 STEP 2507 H=2507:00 +CPU= 12.775
  10:13:19 STEP 2508 H=2508:00 +CPU= 21.219
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002508
  10:13:31 STEP 2509 H=2509:00 +CPU= 11.917
  10:13:43 STEP 2510 H=2510:00 +CPU= 12.261
  10:14:04 STEP 2511 H=2511:00 +CPU= 20.200
  10:14:19 STEP 2512 H=2512:00 +CPU= 13.870
  10:14:31 STEP 2513 H=2513:00 +CPU= 11.758
  10:14:50 STEP 2514 H=2514:00 +CPU= 19.150
  10:15:02 STEP 2515 H=2515:00 +CPU= 12.015
  10:15:16 STEP 2516 H=2516:00 +CPU= 13.824
  10:15:37 STEP 2517 H=2517:00 +CPU= 21.443
  10:15:59 STEP 2518 H=2518:00 +CPU= 16.524
  10:16:14 STEP 2519 H=2519:00 +CPU= 13.381
  10:16:45 STEP 2520 H=2520:00 +CPU= 23.304
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_104.zip
Uploading the intermediate file: upload_file_104.zip
  10:16:57 STEP 2521 H=2521:00 +CPU= 12.080
  10:17:10 STEP 2522 H=2522:00 +CPU= 12.376
  10:17:30 STEP 2523 H=2523:00 +CPU= 19.534
  10:17:51 STEP 2524 H=2524:00 +CPU= 14.555
  10:18:06 STEP 2525 H=2525:00 +CPU= 14.192
  10:18:28 STEP 2526 H=2526:00 +CPU= 21.773
  10:18:42 STEP 2527 H=2527:00 +CPU= 14.249
  10:18:59 STEP 2528 H=2528:00 +CPU= 14.754
  10:19:20 STEP 2529 H=2529:00 +CPU= 21.113
  10:19:32 STEP 2530 H=2530:00 +CPU= 12.355
  10:19:44 STEP 2531 H=2531:00 +CPU= 11.935
  10:20:07 STEP 2532 H=2532:00 +CPU= 22.201
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002532
  10:20:22 STEP 2533 H=2533:00 +CPU= 14.958
  10:20:36 STEP 2534 H=2534:00 +CPU= 13.520
  10:21:01 STEP 2535 H=2535:00 +CPU= 23.656
  10:21:15 STEP 2536 H=2536:00 +CPU= 13.417
  10:21:27 STEP 2537 H=2537:00 +CPU= 12.170
  10:21:47 STEP 2538 H=2538:00 +CPU= 19.277
  10:21:58 STEP 2539 H=2539:00 +CPU= 11.768
  10:22:11 STEP 2540 H=2540:00 +CPU= 12.512
  10:22:32 STEP 2541 H=2541:00 +CPU= 20.433
  10:22:50 STEP 2542 H=2542:00 +CPU= 13.561
  10:23:03 STEP 2543 H=2543:00 +CPU= 12.714
  10:23:44 STEP 2544 H=2544:00 +CPU= 25.964
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_105.zip
Uploading the intermediate file: upload_file_105.zip
  10:24:00 STEP 2545 H=2545:00 +CPU= 13.034
  10:24:20 STEP 2546 H=2546:00 +CPU= 13.157
  10:24:40 STEP 2547 H=2547:00 +CPU= 20.380
  10:24:53 STEP 2548 H=2548:00 +CPU= 12.068
  10:25:05 STEP 2549 H=2549:00 +CPU= 12.609
  10:25:26 STEP 2550 H=2550:00 +CPU= 20.314
  10:25:38 STEP 2551 H=2551:00 +CPU= 12.433
  10:25:51 STEP 2552 H=2552:00 +CPU= 12.812
  10:26:12 STEP 2553 H=2553:00 +CPU= 20.461
  10:26:28 STEP 2554 H=2554:00 +CPU= 16.494
  10:26:48 STEP 2555 H=2555:00 +CPU= 14.932
  10:27:32 STEP 2556 H=2556:00 +CPU= 29.175
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002556
  10:27:51 STEP 2557 H=2557:00 +CPU= 15.624
  10:28:08 STEP 2558 H=2558:00 +CPU= 16.943
  10:28:33 STEP 2559 H=2559:00 +CPU= 22.032
  10:28:48 STEP 2560 H=2560:00 +CPU= 13.797
  10:29:16 STEP 2561 H=2561:00 +CPU= 15.401
  10:29:37 STEP 2562 H=2562:00 +CPU= 20.865
  10:29:57 STEP 2563 H=2563:00 +CPU= 13.415
  10:30:10 STEP 2564 H=2564:00 +CPU= 13.375
  10:30:33 STEP 2565 H=2565:00 +CPU= 21.157
  10:30:46 STEP 2566 H=2566:00 +CPU= 12.495
  10:30:59 STEP 2567 H=2567:00 +CPU= 12.816
  10:31:37 STEP 2568 H=2568:00 +CPU= 27.200
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_106.zip
Uploading the intermediate file: upload_file_106.zip
  10:31:52 STEP 2569 H=2569:00 +CPU= 14.015
  10:32:09 STEP 2570 H=2570:00 +CPU= 14.206
  10:32:29 STEP 2571 H=2571:00 +CPU= 19.585
  10:32:42 STEP 2572 H=2572:00 +CPU= 12.705
  10:32:54 STEP 2573 H=2573:00 +CPU= 12.151
  10:33:14 STEP 2574 H=2574:00 +CPU= 19.357
  10:33:26 STEP 2575 H=2575:00 +CPU= 11.879
  10:33:38 STEP 2576 H=2576:00 +CPU= 12.250
  10:33:57 STEP 2577 H=2577:00 +CPU= 19.259
  10:34:10 STEP 2578 H=2578:00 +CPU= 12.547
  10:34:22 STEP 2579 H=2579:00 +CPU= 12.324
  10:34:54 STEP 2580 H=2580:00 +CPU= 27.912
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002580
  10:35:09 STEP 2581 H=2581:00 +CPU= 13.261
  10:35:22 STEP 2582 H=2582:00 +CPU= 12.756
  10:35:46 STEP 2583 H=2583:00 +CPU= 22.690
  10:36:05 STEP 2584 H=2584:00 +CPU= 17.058
  10:36:21 STEP 2585 H=2585:00 +CPU= 15.290
  10:36:45 STEP 2586 H=2586:00 +CPU= 22.464
  10:36:58 STEP 2587 H=2587:00 +CPU= 13.428
  10:37:11 STEP 2588 H=2588:00 +CPU= 12.668
  10:37:33 STEP 2589 H=2589:00 +CPU= 22.002
  10:37:48 STEP 2590 H=2590:00 +CPU= 14.565
  10:38:02 STEP 2591 H=2591:00 +CPU= 14.447
  10:38:27 STEP 2592 H=2592:00 +CPU= 23.458
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_107.zip
Uploading the intermediate file: upload_file_107.zip
  10:38:42 STEP 2593 H=2593:00 +CPU= 14.109
  10:38:55 STEP 2594 H=2594:00 +CPU= 13.670
  10:39:17 STEP 2595 H=2595:00 +CPU= 21.666
  10:39:32 STEP 2596 H=2596:00 +CPU= 14.358
  10:39:46 STEP 2597 H=2597:00 +CPU= 13.949
  10:40:07 STEP 2598 H=2598:00 +CPU= 21.643
  10:40:20 STEP 2599 H=2599:00 +CPU= 12.275
  10:40:36 STEP 2600 H=2600:00 +CPU= 16.184
  10:41:01 STEP 2601 H=2601:00 +CPU= 25.443
  10:41:18 STEP 2602 H=2602:00 +CPU= 16.580
  10:41:34 STEP 2603 H=2603:00 +CPU= 16.105
  10:42:03 STEP 2604 H=2604:00 +CPU= 28.936
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002604
  10:42:20 STEP 2605 H=2605:00 +CPU= 16.775
  10:42:37 STEP 2606 H=2606:00 +CPU= 16.880
  10:43:04 STEP 2607 H=2607:00 +CPU= 26.802
  10:43:18 STEP 2608 H=2608:00 +CPU= 14.193
  10:43:34 STEP 2609 H=2609:00 +CPU= 15.749
  10:43:58 STEP 2610 H=2610:00 +CPU= 23.932
  10:44:14 STEP 2611 H=2611:00 +CPU= 15.924
  10:44:35 STEP 2612 H=2612:00 +CPU= 18.270
  10:45:16 STEP 2613 H=2613:00 +CPU= 22.197
  10:45:28 STEP 2614 H=2614:00 +CPU= 12.040
  10:45:41 STEP 2615 H=2615:00 +CPU= 12.108
  10:46:05 STEP 2616 H=2616:00 +CPU= 22.433
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194032/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0388_2008050100_123_977_12194032_1_r1194875284_108.zip
  10:46:17 STEP 2617 H=2617:00 +CPU= 11.683
Uploading the intermediate file: upload_file_108.zip
  10:46:30 STEP 2618 H=2618:00 +CPU= 12.588
  10:46:51 STEP 2619 H=2619:00 +CPU= 21.156
  10:47:05 STEP 2620 H=2620:00 +CPU= 13.944
  10:47:18 STEP 2621 H=2621:00 +CPU= 13.057
  10:47:39 STEP 2622 H=2622:00 +CPU= 20.565
  10:47:51 STEP 2623 H=2623:00 +CPU= 12.466
  10:48:05 STEP 2624 H=2624:00 +CPU= 13.932
  10:48:27 STEP 2625 H=2625:00 +CPU= 21.867
  10:48:42 STEP 2626 H=2626:00 +CPU= 14.126
  10:48:54 STEP 2627 H=2627:00 +CPU= 12.556
  10:49:17 STEP 2628 H=2628:00 +CPU= 22.227
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+002628
  10:49:29 STEP 2629 H=2629:00 +CPU= 12.212
  10:49:42 STEP 2630 H=2630:00 +CPU= 12.994
  10:50:01 STEP 2631 H=2631:00 +CPU= 19.345
  10:50:14 STEP 2632 H=2632:00 +CPU= 13.111
  10:50:27 STEP 2633 H=2633:00 +CPU= 12.191
  10:50:46 STEP 2634 H=2634:00 +CPU= 19.842
  10:50:59 STEP 2635 H=2635:00 +CPU= 12.141
  10:51:11 STEP 2636 H=2636:00 +CPU= 12.769
  10:51:31 STEP 2637 H=2637:00 +CPU= 19.604
  10:51:44 STEP 2638 H=2638:00 +CPU= 12.606
  10:51:56 STEP 2639 H=2639:00 +CPU= 12.644
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:37:14 1512026 22301921 oifs_43r3_ps_0388_2008050100_123_977_12194032_1 7,948,800 35,410 0.0045
09 May 2023 16:36:52 1512026 22301921 oifs_43r3_ps_0388_2008050100_123_977_12194032_1 6,134,400 26,817 0.0044
09 May 2023 16:36:44 1512026 22301921 oifs_43r3_ps_0388_2008050100_123_977_12194032_1 5,184,000 22,604 0.0044
09 May 2023 16:36:39 1512026 22301921 oifs_43r3_ps_0388_2008050100_123_977_12194032_1 4,320,000 18,817 0.0044
09 May 2023 16:36:34 1512026 22301921 oifs_43r3_ps_0388_2008050100_123_977_12194032_1 3,456,000 15,121 0.0044
09 May 2023 16:36:29 1512026 22301921 oifs_43r3_ps_0388_2008050100_123_977_12194032_1 2,592,000 11,316 0.0044
09 May 2023 16:36:24 1512026 22301921 oifs_43r3_ps_0388_2008050100_123_977_12194032_1 1,728,000 7,595 0.0044
09 May 2023 16:36:19 1512026 22301921 oifs_43r3_ps_0388_2008050100_123_977_12194032_1 864,000 3,808 0.0044


©2024 cpdn.org