climateprediction.net home page
Task 22291607

Task 22291607

Name oifs_43r3_ps_0443_2019050100_123_988_12205087_0
Workunit 12205087
Created 24 Dec 2022, 23:17:42 UTC
Sent 21 Jan 2023, 20:31:27 UTC
Report deadline 22 Mar 2023, 20:31:27 UTC
Received 24 Jan 2023, 23:33:47 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1467890
Run time 1 days 0 hours 32 min 53 sec
CPU time 23 hours 32 min 33 sec
Validate state Invalid
Credit 11,370.24
Device peak FLOPS 4.08 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.15 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,670.81 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
PU= 27.990
  06:04:00 STEP 2282 H=2282:00 +CPU= 26.542
  06:04:39 STEP 2283 H=2283:00 +CPU= 39.140
  06:05:04 STEP 2284 H=2284:00 +CPU= 25.410
  06:05:30 STEP 2285 H=2285:00 +CPU= 25.461
  06:06:13 STEP 2286 H=2286:00 +CPU= 43.193
  06:06:43 STEP 2287 H=2287:00 +CPU= 29.619
  06:07:12 STEP 2288 H=2288:00 +CPU= 28.837
  06:07:52 STEP 2289 H=2289:00 +CPU= 39.930
  06:08:18 STEP 2290 H=2290:00 +CPU= 25.782
  06:08:42 STEP 2291 H=2291:00 +CPU= 24.639
  06:09:27 STEP 2292 H=2292:00 +CPU= 44.368
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002292
  06:09:52 STEP 2293 H=2293:00 +CPU= 25.067
  06:10:17 STEP 2294 H=2294:00 +CPU= 25.550
  06:10:57 STEP 2295 H=2295:00 +CPU= 39.429
  06:11:24 STEP 2296 H=2296:00 +CPU= 27.010
  06:11:54 STEP 2297 H=2297:00 +CPU= 29.603
  06:12:36 STEP 2298 H=2298:00 +CPU= 42.694
  06:13:01 STEP 2299 H=2299:00 +CPU= 24.972
  06:13:27 STEP 2300 H=2300:00 +CPU= 25.460
  06:14:07 STEP 2301 H=2301:00 +CPU= 40.231
  06:14:32 STEP 2302 H=2302:00 +CPU= 25.243
  06:14:58 STEP 2303 H=2303:00 +CPU= 25.860
  06:15:51 STEP 2304 H=2304:00 +CPU= 52.444
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_95.zip
Uploading the intermediate file: upload_file_95.zip
  06:16:15 STEP 2305 H=2305:00 +CPU= 24.488
  06:16:44 STEP 2306 H=2306:00 +CPU= 28.788
  06:17:23 STEP 2307 H=2307:00 +CPU= 38.781
  06:17:48 STEP 2308 H=2308:00 +CPU= 24.995
  06:18:13 STEP 2309 H=2309:00 +CPU= 25.159
  06:19:02 STEP 2310 H=2310:00 +CPU= 48.394
  06:19:27 STEP 2311 H=2311:00 +CPU= 25.097
  06:19:52 STEP 2312 H=2312:00 +CPU= 24.908
  06:20:31 STEP 2313 H=2313:00 +CPU= 39.327
  06:20:56 STEP 2314 H=2314:00 +CPU= 25.156
  06:21:21 STEP 2315 H=2315:00 +CPU= 24.156
  06:22:07 STEP 2316 H=2316:00 +CPU= 46.568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002316
  06:22:37 STEP 2317 H=2317:00 +CPU= 29.648
  06:23:01 STEP 2318 H=2318:00 +CPU= 24.451
  06:23:41 STEP 2319 H=2319:00 +CPU= 39.155
  06:24:06 STEP 2320 H=2320:00 +CPU= 24.839
  06:24:30 STEP 2321 H=2321:00 +CPU= 24.764
  06:25:09 STEP 2322 H=2322:00 +CPU= 39.103
  06:25:37 STEP 2323 H=2323:00 +CPU= 27.540
  06:26:03 STEP 2324 H=2324:00 +CPU= 25.821
  06:26:48 STEP 2325 H=2325:00 +CPU= 45.105
  06:27:13 STEP 2326 H=2326:00 +CPU= 24.762
  06:27:44 STEP 2327 H=2327:00 +CPU= 30.566
  06:28:34 STEP 2328 H=2328:00 +CPU= 50.112
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_96.zip
Uploading the intermediate file: upload_file_96.zip
  06:29:03 STEP 2329 H=2329:00 +CPU= 29.266
  06:29:29 STEP 2330 H=2330:00 +CPU= 26.319
  06:30:12 STEP 2331 H=2331:00 +CPU= 42.283
  06:30:37 STEP 2332 H=2332:00 +CPU= 25.376
  06:31:03 STEP 2333 H=2333:00 +CPU= 25.908
  06:31:47 STEP 2334 H=2334:00 +CPU= 43.978
  06:32:12 STEP 2335 H=2335:00 +CPU= 24.573
  06:32:37 STEP 2336 H=2336:00 +CPU= 24.948
  06:33:16 STEP 2337 H=2337:00 +CPU= 39.034
  06:33:46 STEP 2338 H=2338:00 +CPU= 30.032
  06:34:12 STEP 2339 H=2339:00 +CPU= 26.289
  06:34:56 STEP 2340 H=2340:00 +CPU= 43.337
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002340
  06:35:20 STEP 2341 H=2341:00 +CPU= 24.760
  06:35:46 STEP 2342 H=2342:00 +CPU= 25.851
  06:36:26 STEP 2343 H=2343:00 +CPU= 39.597
  06:36:52 STEP 2344 H=2344:00 +CPU= 25.690
  06:37:17 STEP 2345 H=2345:00 +CPU= 24.917
  06:37:58 STEP 2346 H=2346:00 +CPU= 40.874
  06:38:23 STEP 2347 H=2347:00 +CPU= 25.270
  06:38:54 STEP 2348 H=2348:00 +CPU= 30.533
  06:39:41 STEP 2349 H=2349:00 +CPU= 47.336
  06:40:07 STEP 2350 H=2350:00 +CPU= 26.371
  06:40:33 STEP 2351 H=2351:00 +CPU= 25.062
  06:41:18 STEP 2352 H=2352:00 +CPU= 45.477
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_97.zip
Uploading the intermediate file: upload_file_97.zip
  06:41:43 STEP 2353 H=2353:00 +CPU= 25.013
  06:42:09 STEP 2354 H=2354:00 +CPU= 25.443
  06:42:53 STEP 2355 H=2355:00 +CPU= 44.582
  06:43:22 STEP 2356 H=2356:00 +CPU= 28.507
  06:43:53 STEP 2357 H=2357:00 +CPU= 30.733
  06:44:42 STEP 2358 H=2358:00 +CPU= 48.847
  06:45:06 STEP 2359 H=2359:00 +CPU= 24.306
  06:45:32 STEP 2360 H=2360:00 +CPU= 25.602
  06:46:11 STEP 2361 H=2361:00 +CPU= 39.473
  06:46:37 STEP 2362 H=2362:00 +CPU= 25.330
  06:47:06 STEP 2363 H=2363:00 +CPU= 29.492
  06:47:52 STEP 2364 H=2364:00 +CPU= 45.699
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002364
  06:48:16 STEP 2365 H=2365:00 +CPU= 24.095
  06:48:41 STEP 2366 H=2366:00 +CPU= 24.930
  06:49:20 STEP 2367 H=2367:00 +CPU= 38.775
  06:49:45 STEP 2368 H=2368:00 +CPU= 25.048
  06:50:10 STEP 2369 H=2369:00 +CPU= 24.585
  06:50:50 STEP 2370 H=2370:00 +CPU= 40.000
  06:51:17 STEP 2371 H=2371:00 +CPU= 26.761
  06:51:47 STEP 2372 H=2372:00 +CPU= 30.521
  06:52:28 STEP 2373 H=2373:00 +CPU= 40.612
  06:52:53 STEP 2374 H=2374:00 +CPU= 25.185
  06:53:19 STEP 2375 H=2375:00 +CPU= 25.671
  06:54:05 STEP 2376 H=2376:00 +CPU= 45.833
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_98.zip
Uploading the intermediate file: upload_file_98.zip
  06:54:29 STEP 2377 H=2377:00 +CPU= 24.087
  06:54:54 STEP 2378 H=2378:00 +CPU= 25.479
  06:55:34 STEP 2379 H=2379:00 +CPU= 39.371
  06:55:59 STEP 2380 H=2380:00 +CPU= 25.644
  06:56:25 STEP 2381 H=2381:00 +CPU= 25.104
  06:57:13 STEP 2382 H=2382:00 +CPU= 48.203
  06:57:37 STEP 2383 H=2383:00 +CPU= 24.050
  06:58:02 STEP 2384 H=2384:00 +CPU= 25.495
  06:58:42 STEP 2385 H=2385:00 +CPU= 39.157
  06:59:07 STEP 2386 H=2386:00 +CPU= 25.303
  06:59:32 STEP 2387 H=2387:00 +CPU= 24.814
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0443
(argv4) batchid: 988
(argv5) wuid: 12205087
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fa4be3f35fa890eb873061196ef34ec2 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0443_2019050100_123_988_12205087.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0443_2019050100_123_988_12205087.zip
ic_ancil_file: ic_ancil_12205087
ifsdata_file: ifsdata_12205087
climate_data_file: clim_data_12205087
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1ef312cc01726339463e13ad7edb7fb2 to: /var/lib/boinc-client/slots/1/ic_ancil_12205087.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12205087.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f5e710b9c9b2b7877f27758e527a020f to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12205087.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12205087.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12205087.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12205087.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12205087.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12205087.xml
-- Model is restarting --
Adjusting last_iter, 2388, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12205087.xml
last_cpu_time: 76527
upload_file_number: 99
last_iter: 2375
last_upload: 8553600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439
The child process has been launched with process id: 14824
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:14824:14824] [20230124:213013:1674592213:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:31:41 STEP 2376 H=2376:00 +CPU= 37.942
  21:32:02 STEP 2377 H=2377:00 +CPU= 20.358
  21:32:23 STEP 2378 H=2378:00 +CPU= 20.990
  21:32:56 STEP 2379 H=2379:00 +CPU= 32.946
  21:33:18 STEP 2380 H=2380:00 +CPU= 21.110
  21:33:38 STEP 2381 H=2381:00 +CPU= 20.445
  21:34:13 STEP 2382 H=2382:00 +CPU= 35.184
  21:34:34 STEP 2383 H=2383:00 +CPU= 20.459
  21:34:55 STEP 2384 H=2384:00 +CPU= 21.302
  21:35:31 STEP 2385 H=2385:00 +CPU= 35.771
  21:35:54 STEP 2386 H=2386:00 +CPU= 22.753
  21:36:15 STEP 2387 H=2387:00 +CPU= 20.600
  21:36:54 STEP 2388 H=2388:00 +CPU= 39.336
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002388
  21:37:20 STEP 2389 H=2389:00 +CPU= 23.341
  21:37:43 STEP 2390 H=2390:00 +CPU= 23.111
  21:38:19 STEP 2391 H=2391:00 +CPU= 35.485
  21:38:45 STEP 2392 H=2392:00 +CPU= 26.135
  21:39:09 STEP 2393 H=2393:00 +CPU= 23.767
  21:39:43 STEP 2394 H=2394:00 +CPU= 34.013
  21:40:06 STEP 2395 H=2395:00 +CPU= 22.452
  21:40:32 STEP 2396 H=2396:00 +CPU= 24.452
  21:41:10 STEP 2397 H=2397:00 +CPU= 37.970
  21:41:34 STEP 2398 H=2398:00 +CPU= 24.231
  21:41:59 STEP 2399 H=2399:00 +CPU= 24.664
  21:42:51 STEP 2400 H=2400:00 +CPU= 44.368
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_99.zip
Uploading the intermediate file: upload_file_99.zip
  21:43:14 STEP 2401 H=2401:00 +CPU= 22.940
  21:43:39 STEP 2402 H=2402:00 +CPU= 24.848
  21:44:18 STEP 2403 H=2403:00 +CPU= 39.127
  21:44:43 STEP 2404 H=2404:00 +CPU= 24.755
  21:45:07 STEP 2405 H=2405:00 +CPU= 23.440
  21:45:47 STEP 2406 H=2406:00 +CPU= 39.810
  21:46:11 STEP 2407 H=2407:00 +CPU= 23.973
  21:46:36 STEP 2408 H=2408:00 +CPU= 25.521
  21:47:16 STEP 2409 H=2409:00 +CPU= 39.373
  21:47:41 STEP 2410 H=2410:00 +CPU= 24.780
  21:48:09 STEP 2411 H=2411:00 +CPU= 28.708
  21:48:54 STEP 2412 H=2412:00 +CPU= 44.137
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002412
  21:49:18 STEP 2413 H=2413:00 +CPU= 24.354
  21:49:46 STEP 2414 H=2414:00 +CPU= 27.416
  21:50:24 STEP 2415 H=2415:00 +CPU= 38.749
  21:50:52 STEP 2416 H=2416:00 +CPU= 27.286
  21:51:22 STEP 2417 H=2417:00 +CPU= 30.145
  21:52:04 STEP 2418 H=2418:00 +CPU= 42.233
  21:52:33 STEP 2419 H=2419:00 +CPU= 28.387
  21:52:57 STEP 2420 H=2420:00 +CPU= 23.661
  21:53:36 STEP 2421 H=2421:00 +CPU= 39.545
  21:54:02 STEP 2422 H=2422:00 +CPU= 25.345
  21:54:27 STEP 2423 H=2423:00 +CPU= 25.221
  21:55:16 STEP 2424 H=2424:00 +CPU= 47.833
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_100.zip
Uploading the intermediate file: upload_file_100.zip
  21:55:42 STEP 2425 H=2425:00 +CPU= 25.990
  21:56:07 STEP 2426 H=2426:00 +CPU= 25.153
  21:56:51 STEP 2427 H=2427:00 +CPU= 43.689
  21:57:16 STEP 2428 H=2428:00 +CPU= 24.730
  21:57:42 STEP 2429 H=2429:00 +CPU= 25.754
  21:58:23 STEP 2430 H=2430:00 +CPU= 41.051
  21:58:48 STEP 2431 H=2431:00 +CPU= 24.386
  21:59:18 STEP 2432 H=2432:00 +CPU= 29.988
  22:00:04 STEP 2433 H=2433:00 +CPU= 46.243
  22:00:30 STEP 2434 H=2434:00 +CPU= 25.903
  22:00:57 STEP 2435 H=2435:00 +CPU= 26.236
  22:01:40 STEP 2436 H=2436:00 +CPU= 42.874
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002436
  22:02:10 STEP 2437 H=2437:00 +CPU= 30.254
  22:02:35 STEP 2438 H=2438:00 +CPU= 24.534
  22:03:17 STEP 2439 H=2439:00 +CPU= 42.330
  22:03:48 STEP 2440 H=2440:00 +CPU= 30.551
  22:04:13 STEP 2441 H=2441:00 +CPU= 25.580
  22:04:55 STEP 2442 H=2442:00 +CPU= 41.536
  22:05:20 STEP 2443 H=2443:00 +CPU= 24.888
  22:05:46 STEP 2444 H=2444:00 +CPU= 25.846
  22:06:31 STEP 2445 H=2445:00 +CPU= 44.133
  22:06:56 STEP 2446 H=2446:00 +CPU= 25.128
  22:07:25 STEP 2447 H=2447:00 +CPU= 29.508
  22:08:17 STEP 2448 H=2448:00 +CPU= 49.825
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_101.zip
Uploading the intermediate file: upload_file_101.zip
  22:08:44 STEP 2449 H=2449:00 +CPU= 26.890
  22:09:11 STEP 2450 H=2450:00 +CPU= 26.775
  22:09:50 STEP 2451 H=2451:00 +CPU= 39.284
  22:10:20 STEP 2452 H=2452:00 +CPU= 29.596
  22:10:50 STEP 2453 H=2453:00 +CPU= 29.761
  22:11:30 STEP 2454 H=2454:00 +CPU= 40.603
  22:11:59 STEP 2455 H=2455:00 +CPU= 25.533
  22:12:24 STEP 2456 H=2456:00 +CPU= 25.318
  22:13:08 STEP 2457 H=2457:00 +CPU= 44.285
  22:13:34 STEP 2458 H=2458:00 +CPU= 25.716
  22:13:59 STEP 2459 H=2459:00 +CPU= 24.271
  22:14:50 STEP 2460 H=2460:00 +CPU= 51.528
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002460
  22:15:18 STEP 2461 H=2461:00 +CPU= 27.229
  22:15:43 STEP 2462 H=2462:00 +CPU= 25.467
  22:16:22 STEP 2463 H=2463:00 +CPU= 39.206
  22:16:50 STEP 2464 H=2464:00 +CPU= 27.733
  22:17:20 STEP 2465 H=2465:00 +CPU= 29.884
  22:18:09 STEP 2466 H=2466:00 +CPU= 48.550
  22:18:35 STEP 2467 H=2467:00 +CPU= 25.773
  22:19:00 STEP 2468 H=2468:00 +CPU= 24.958
  22:19:43 STEP 2469 H=2469:00 +CPU= 43.202
  22:20:08 STEP 2470 H=2470:00 +CPU= 25.314
  22:20:34 STEP 2471 H=2471:00 +CPU= 25.526
  22:21:30 STEP 2472 H=2472:00 +CPU= 52.807
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_102.zip
Uploading the intermediate file: upload_file_102.zip
  22:21:54 STEP 2473 H=2473:00 +CPU= 24.067
  22:22:23 STEP 2474 H=2474:00 +CPU= 28.705
  22:23:09 STEP 2475 H=2475:00 +CPU= 45.999
  22:23:36 STEP 2476 H=2476:00 +CPU= 26.970
  22:24:04 STEP 2477 H=2477:00 +CPU= 27.633
  22:24:44 STEP 2478 H=2478:00 +CPU= 39.692
  22:25:13 STEP 2479 H=2479:00 +CPU= 29.108
  22:25:43 STEP 2480 H=2480:00 +CPU= 30.457
  22:26:32 STEP 2481 H=2481:00 +CPU= 48.127
  22:26:57 STEP 2482 H=2482:00 +CPU= 25.252
  22:27:22 STEP 2483 H=2483:00 +CPU= 24.733
  22:28:12 STEP 2484 H=2484:00 +CPU= 49.877
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002484
  22:28:38 STEP 2485 H=2485:00 +CPU= 26.158
  22:29:04 STEP 2486 H=2486:00 +CPU= 26.471
  22:29:49 STEP 2487 H=2487:00 +CPU= 44.668
  22:30:15 STEP 2488 H=2488:00 +CPU= 25.631
  22:30:45 STEP 2489 H=2489:00 +CPU= 29.689
  22:31:24 STEP 2490 H=2490:00 +CPU= 38.970
  22:31:50 STEP 2491 H=2491:00 +CPU= 26.425
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0443
(argv4) batchid: 988
(argv5) wuid: 12205087
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fa4be3f35fa890eb873061196ef34ec2 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0443_2019050100_123_988_12205087.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0443_2019050100_123_988_12205087.zip
ic_ancil_file: ic_ancil_12205087
ifsdata_file: ifsdata_12205087
climate_data_file: clim_data_12205087
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1ef312cc01726339463e13ad7edb7fb2 to: /var/lib/boinc-client/slots/1/ic_ancil_12205087.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12205087.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f5e710b9c9b2b7877f27758e527a020f to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12205087.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12205087.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12205087.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12205087.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12205087.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12205087.xml
-- Model is restarting --
Adjusting last_iter, 2492, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12205087.xml
last_cpu_time: 80146
upload_file_number: 103
last_iter: 2471
last_upload: 8899200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439
The child process has been launched with process id: 15600
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:15600:15600] [20230124:223305:1674595985:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:34:09 STEP 2472 H=2472:00 +CPU= 54.159
  22:34:35 STEP 2473 H=2473:00 +CPU= 25.739
  22:35:06 STEP 2474 H=2474:00 +CPU= 30.669
  22:35:47 STEP 2475 H=2475:00 +CPU= 41.702
  22:36:17 STEP 2476 H=2476:00 +CPU= 29.289
  22:36:43 STEP 2477 H=2477:00 +CPU= 26.151
  22:37:24 STEP 2478 H=2478:00 +CPU= 40.646
  22:37:55 STEP 2479 H=2479:00 +CPU= 30.483
  22:38:20 STEP 2480 H=2480:00 +CPU= 25.271
  22:39:04 STEP 2481 H=2481:00 +CPU= 43.506
  22:39:31 STEP 2482 H=2482:00 +CPU= 27.680
  22:39:56 STEP 2483 H=2483:00 +CPU= 24.759
  22:40:48 STEP 2484 H=2484:00 +CPU= 51.388
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002484
  22:41:13 STEP 2485 H=2485:00 +CPU= 25.030
  22:41:42 STEP 2486 H=2486:00 +CPU= 29.169
  22:42:21 STEP 2487 H=2487:00 +CPU= 38.859
  22:42:47 STEP 2488 H=2488:00 +CPU= 25.599
  22:43:18 STEP 2489 H=2489:00 +CPU= 30.697
  22:44:06 STEP 2490 H=2490:00 +CPU= 48.550
  22:44:35 STEP 2491 H=2491:00 +CPU= 28.167
  22:44:59 STEP 2492 H=2492:00 +CPU= 24.725
  22:45:44 STEP 2493 H=2493:00 +CPU= 43.833
  22:46:12 STEP 2494 H=2494:00 +CPU= 28.024
  22:46:37 STEP 2495 H=2495:00 +CPU= 25.413
  22:47:33 STEP 2496 H=2496:00 +CPU= 54.707
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_103.zip
Uploading the intermediate file: upload_file_103.zip
  22:47:57 STEP 2497 H=2497:00 +CPU= 24.456
  22:48:27 STEP 2498 H=2498:00 +CPU= 29.977
  22:49:10 STEP 2499 H=2499:00 +CPU= 42.105
  22:49:37 STEP 2500 H=2500:00 +CPU= 27.811
  22:50:08 STEP 2501 H=2501:00 +CPU= 30.192
  22:50:57 STEP 2502 H=2502:00 +CPU= 49.338
  22:51:23 STEP 2503 H=2503:00 +CPU= 26.200
  22:51:49 STEP 2504 H=2504:00 +CPU= 25.569
  22:52:34 STEP 2505 H=2505:00 +CPU= 44.397
  22:53:01 STEP 2506 H=2506:00 +CPU= 27.239
  22:53:28 STEP 2507 H=2507:00 +CPU= 26.750
  22:54:17 STEP 2508 H=2508:00 +CPU= 49.572
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002508
  22:54:42 STEP 2509 H=2509:00 +CPU= 24.161
  22:55:11 STEP 2510 H=2510:00 +CPU= 29.497
  22:55:58 STEP 2511 H=2511:00 +CPU= 46.648
  22:56:26 STEP 2512 H=2512:00 +CPU= 27.652
  22:56:56 STEP 2513 H=2513:00 +CPU= 30.020
  22:57:41 STEP 2514 H=2514:00 +CPU= 45.378
  22:58:12 STEP 2515 H=2515:00 +CPU= 30.344
  22:58:36 STEP 2516 H=2516:00 +CPU= 24.007
  22:59:20 STEP 2517 H=2517:00 +CPU= 44.035
  22:59:47 STEP 2518 H=2518:00 +CPU= 27.001
  23:00:12 STEP 2519 H=2519:00 +CPU= 25.377
  23:01:08 STEP 2520 H=2520:00 +CPU= 55.653
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_104.zip
Uploading the intermediate file: upload_file_104.zip
  23:01:38 STEP 2521 H=2521:00 +CPU= 29.618
  23:02:10 STEP 2522 H=2522:00 +CPU= 31.600
  23:02:48 STEP 2523 H=2523:00 +CPU= 38.295
  23:03:17 STEP 2524 H=2524:00 +CPU= 29.150
  23:03:47 STEP 2525 H=2525:00 +CPU= 29.998
  23:04:33 STEP 2526 H=2526:00 +CPU= 45.424
  23:05:02 STEP 2527 H=2527:00 +CPU= 29.363
  23:05:27 STEP 2528 H=2528:00 +CPU= 24.671
  23:06:12 STEP 2529 H=2529:00 +CPU= 44.635
  23:06:43 STEP 2530 H=2530:00 +CPU= 30.683
  23:07:13 STEP 2531 H=2531:00 +CPU= 29.941
  23:08:00 STEP 2532 H=2532:00 +CPU= 46.814
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002532
  23:08:24 STEP 2533 H=2533:00 +CPU= 24.765
  23:08:54 STEP 2534 H=2534:00 +CPU= 29.446
  23:09:36 STEP 2535 H=2535:00 +CPU= 42.078
  23:10:05 STEP 2536 H=2536:00 +CPU= 28.452
  23:10:35 STEP 2537 H=2537:00 +CPU= 30.250
  23:11:20 STEP 2538 H=2538:00 +CPU= 45.002
  23:11:51 STEP 2539 H=2539:00 +CPU= 30.477
  23:12:15 STEP 2540 H=2540:00 +CPU= 24.601
  23:12:59 STEP 2541 H=2541:00 +CPU= 43.788
  23:13:29 STEP 2542 H=2542:00 +CPU= 29.787
  23:13:56 STEP 2543 H=2543:00 +CPU= 26.706
  23:14:49 STEP 2544 H=2544:00 +CPU= 52.404
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_105.zip
Uploading the intermediate file: upload_file_105.zip
  23:15:14 STEP 2545 H=2545:00 +CPU= 24.821
  23:15:44 STEP 2546 H=2546:00 +CPU= 29.219
  23:16:28 STEP 2547 H=2547:00 +CPU= 43.981
  23:16:56 STEP 2548 H=2548:00 +CPU= 28.279
  23:17:26 STEP 2549 H=2549:00 +CPU= 30.229
  23:18:06 STEP 2550 H=2550:00 +CPU= 39.629
  23:18:37 STEP 2551 H=2551:00 +CPU= 30.626
  23:19:05 STEP 2552 H=2552:00 +CPU= 28.106
  23:19:49 STEP 2553 H=2553:00 +CPU= 43.786
  23:20:20 STEP 2554 H=2554:00 +CPU= 30.847
  23:20:50 STEP 2555 H=2555:00 +CPU= 29.884
  23:21:38 STEP 2556 H=2556:00 +CPU= 48.266
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002556
  23:22:04 STEP 2557 H=2557:00 +CPU= 25.546
  23:22:36 STEP 2558 H=2558:00 +CPU= 31.727
  23:23:17 STEP 2559 H=2559:00 +CPU= 41.074
  23:23:43 STEP 2560 H=2560:00 +CPU= 26.641
  23:24:14 STEP 2561 H=2561:00 +CPU= 30.661
  23:25:01 STEP 2562 H=2562:00 +CPU= 46.347
  23:25:31 STEP 2563 H=2563:00 +CPU= 30.050
  23:26:02 STEP 2564 H=2564:00 +CPU= 30.646
  23:26:46 STEP 2565 H=2565:00 +CPU= 44.547
  23:27:12 STEP 2566 H=2566:00 +CPU= 25.370
  23:27:38 STEP 2567 H=2567:00 +CPU= 25.552
  23:28:32 STEP 2568 H=2568:00 +CPU= 53.366
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_106.zip
Uploading the intermediate file: upload_file_106.zip
  23:28:57 STEP 2569 H=2569:00 +CPU= 24.342
  23:29:26 STEP 2570 H=2570:00 +CPU= 29.503
  23:30:16 STEP 2571 H=2571:00 +CPU= 49.234
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:30:53 STEP 2572 H=2572:00 +CPU= 26.393
  23:31:23 STEP 2573 H=2573:00 +CPU= 29.896
  23:32:04 STEP 2574 H=2574:00 +CPU= 40.716
  23:32:34 STEP 2575 H=2575:00 +CPU= 30.267
  23:33:02 STEP 2576 H=2576:00 +CPU= 27.267
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0443
(argv4) batchid: 988
(argv5) wuid: 12205087
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fa4be3f35fa890eb873061196ef34ec2 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0443_2019050100_123_988_12205087.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0443_2019050100_123_988_12205087.zip
ic_ancil_file: ic_ancil_12205087
ifsdata_file: ifsdata_12205087
climate_data_file: clim_data_12205087
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1ef312cc01726339463e13ad7edb7fb2 to: /var/lib/boinc-client/slots/1/ic_ancil_12205087.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12205087.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f5e710b9c9b2b7877f27758e527a020f to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12205087.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12205087.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12205087.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12205087.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12205087.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12205087.xml
-- Model is restarting --
Adjusting last_iter, 2577, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12205087.xml
last_cpu_time: 83738
upload_file_number: 107
last_iter: 2567
last_upload: 9244800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439
The child process has been launched with process id: 16357
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:16357:16357] [20230124:233442:1674599682:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:35:38 STEP 2568 H=2568:00 +CPU= 44.842
  23:36:03 STEP 2569 H=2569:00 +CPU= 25.314
  23:36:35 STEP 2570 H=2570:00 +CPU= 31.611
  23:37:21 STEP 2571 H=2571:00 +CPU= 45.676
  23:37:51 STEP 2572 H=2572:00 +CPU= 29.816
  23:38:21 STEP 2573 H=2573:00 +CPU= 29.737
  23:39:04 STEP 2574 H=2574:00 +CPU= 42.578
  23:39:33 STEP 2575 H=2575:00 +CPU= 29.651
  23:39:58 STEP 2576 H=2576:00 +CPU= 24.272
  23:40:45 STEP 2577 H=2577:00 +CPU= 47.661
  23:41:11 STEP 2578 H=2578:00 +CPU= 25.887
  23:41:40 STEP 2579 H=2579:00 +CPU= 28.944
  23:42:26 STEP 2580 H=2580:00 +CPU= 45.637
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002580
  23:42:53 STEP 2581 H=2581:00 +CPU= 27.260
  23:43:25 STEP 2582 H=2582:00 +CPU= 31.195
  23:44:13 STEP 2583 H=2583:00 +CPU= 47.799
  23:44:41 STEP 2584 H=2584:00 +CPU= 28.557
  23:45:05 STEP 2585 H=2585:00 +CPU= 24.024
  23:45:51 STEP 2586 H=2586:00 +CPU= 45.563
  23:46:21 STEP 2587 H=2587:00 +CPU= 29.827
  23:46:47 STEP 2588 H=2588:00 +CPU= 26.126
  23:47:35 STEP 2589 H=2589:00 +CPU= 47.912
  23:48:06 STEP 2590 H=2590:00 +CPU= 30.501
  23:48:38 STEP 2591 H=2591:00 +CPU= 31.397
  23:49:24 STEP 2592 H=2592:00 +CPU= 45.974
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205087/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0443_2019050100_123_988_12205087_0_r1855259439_107.zip
Uploading the intermediate file: upload_file_107.zip
  23:49:54 STEP 2593 H=2593:00 +CPU= 29.888
  23:50:25 STEP 2594 H=2594:00 +CPU= 30.733
  23:51:10 STEP 2595 H=2595:00 +CPU= 45.171
  23:51:41 STEP 2596 H=2596:00 +CPU= 30.737
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:38:51 1467890 22291607 oifs_43r3_ps_0443_2019050100_123_988_12205087_0 8,121,600 72,372 0.0089
09 May 2023 16:38:43 1467890 22291607 oifs_43r3_ps_0443_2019050100_123_988_12205087_0 7,257,600 64,772 0.0089
09 May 2023 16:38:36 1467890 22291607 oifs_43r3_ps_0443_2019050100_123_988_12205087_0 6,393,600 57,183 0.0089
09 May 2023 16:38:28 1467890 22291607 oifs_43r3_ps_0443_2019050100_123_988_12205087_0 5,529,600 49,645 0.0090
09 May 2023 16:37:02 1467890 22291607 oifs_43r3_ps_0443_2019050100_123_988_12205087_0 4,320,000 39,261 0.0091
09 May 2023 16:36:52 1467890 22291607 oifs_43r3_ps_0443_2019050100_123_988_12205087_0 3,456,000 31,452 0.0091
09 May 2023 16:36:42 1467890 22291607 oifs_43r3_ps_0443_2019050100_123_988_12205087_0 2,592,000 23,623 0.0091
09 May 2023 16:36:32 1467890 22291607 oifs_43r3_ps_0443_2019050100_123_988_12205087_0 1,728,000 15,815 0.0092


©2024 cpdn.org