<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
[process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2220640:2220640] [20230115:094139:1673743299:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:44:11 STEP 312 H= 312:00 +CPU= 36.873
09:44:32 STEP 313 H= 313:00 +CPU= 20.971
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:47:26 STEP 314 H= 314:00 +CPU= 21.451
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:49:40 STEP 315 H= 315:00 +CPU= 32.608
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:52:12 STEP 316 H= 316:00 +CPU= 21.310
09:52:34 STEP 317 H= 317:00 +CPU= 21.049
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:55:28 STEP 318 H= 318:00 +CPU= 33.158
09:55:49 STEP 319 H= 319:00 +CPU= 21.180
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:58:12 STEP 320 H= 320:00 +CPU= 21.305
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:00:56 STEP 321 H= 321:00 +CPU= 32.914
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:02:49 STEP 322 H= 322:00 +CPU= 21.922
10:03:10 STEP 323 H= 323:00 +CPU= 20.539
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:05:47 STEP 324 H= 324:00 +CPU= 36.059
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000324
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000324
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000324
10:06:07 STEP 325 H= 325:00 +CPU= 20.280
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:08:20 STEP 326 H= 326:00 +CPU= 21.004
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:10:54 STEP 327 H= 327:00 +CPU= 32.848
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:12:46 STEP 328 H= 328:00 +CPU= 21.152
10:13:07 STEP 329 H= 329:00 +CPU= 20.601
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:15:21 STEP 330 H= 330:00 +CPU= 33.398
10:15:41 STEP 331 H= 331:00 +CPU= 19.956
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:18:35 STEP 332 H= 332:00 +CPU= 21.381
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:20:59 STEP 333 H= 333:00 +CPU= 33.168
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:23:01 STEP 334 H= 334:00 +CPU= 20.767
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:25:24 STEP 335 H= 335:00 +CPU= 21.141
10:26:02 STEP 336 H= 336:00 +CPU= 38.383
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000336
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000336
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000324
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0470_2011050100_123_980_12197114_0_r1819580094_13.zip
Uploading the intermediate file: upload_file_13.zip
10:28:34 STEP 337 H= 337:00 +CPU= 20.945
10:28:56 STEP 338 H= 338:00 +CPU= 21.293
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:31:29 STEP 339 H= 339:00 +CPU= 33.029
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:33:41 STEP 340 H= 340:00 +CPU= 21.171
10:34:03 STEP 341 H= 341:00 +CPU= 21.263
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:36:48 STEP 342 H= 342:00 +CPU= 33.474
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:39:00 STEP 343 H= 343:00 +CPU= 20.686
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:40:33 STEP 344 H= 344:00 +CPU= 21.349
10:41:06 STEP 345 H= 345:00 +CPU= 32.796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:43:28 STEP 346 H= 346:00 +CPU= 21.124
10:43:49 STEP 347 H= 347:00 +CPU= 20.166
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:46:26 STEP 348 H= 348:00 +CPU= 36.478
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000348
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:49:28 STEP 349 H= 349:00 +CPU= 20.102
10:49:49 STEP 350 H= 350:00 +CPU= 21.281
10:50:21 STEP 351 H= 351:00 +CPU= 32.286
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:54:55 STEP 352 H= 352:00 +CPU= 21.175
10:55:15 STEP 353 H= 353:00 +CPU= 20.364
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:58:40 STEP 354 H= 354:00 +CPU= 33.869
10:59:01 STEP 355 H= 355:00 +CPU= 20.740
10:59:22 STEP 356 H= 356:00 +CPU= 20.399
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:02:45 STEP 357 H= 357:00 +CPU= 32.329
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:05:08 STEP 358 H= 358:00 +CPU= 21.255
11:05:29 STEP 359 H= 359:00 +CPU= 21.386
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:08:40 STEP 360 H= 360:00 +CPU= 38.869
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000348
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0470_2011050100_123_980_12197114_0_r1819580094_14.zip
Uploading the intermediate file: upload_file_14.zip
11:09:00 STEP 361 H= 361:00 +CPU= 20.273
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:12:03 STEP 362 H= 362:00 +CPU= 20.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:15:07 STEP 363 H= 363:00 +CPU= 33.150
11:15:28 STEP 364 H= 364:00 +CPU= 21.113
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:17:51 STEP 365 H= 365:00 +CPU= 20.288
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:21:27 STEP 366 H= 366:00 +CPU= 33.516
11:21:47 STEP 367 H= 367:00 +CPU= 20.364
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:25:00 STEP 368 H= 368:00 +CPU= 21.167
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:28:05 STEP 369 H= 369:00 +CPU= 32.959
11:28:26 STEP 370 H= 370:00 +CPU= 21.625
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:30:58 STEP 371 H= 371:00 +CPU= 20.468
11:31:34 STEP 372 H= 372:00 +CPU= 35.416
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000372
11:35:15 STEP 373 H= 373:00 +CPU= 20.701
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:37:57 STEP 374 H= 374:00 +CPU= 20.456
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:40:13 STEP 375 H= 375:00 +CPU= 33.288
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:42:36 STEP 376 H= 376:00 +CPU= 20.790
11:42:56 STEP 377 H= 377:00 +CPU= 20.661
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2011050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0470
(argv4) batchid: 980
(argv5) wuid: 12197114
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_800066ac9e16545a34d5ff2406f1f5a8 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0470_2011050100_123_980_12197114.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0470_2011050100_123_980_12197114.zip
ic_ancil_file: ic_ancil_12197114
ifsdata_file: ifsdata_12197114
climate_data_file: clim_data_12197114
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8c8abd1e077c18228d5f79e83b032356 to: /var/lib/boinc-client/slots/4/ic_ancil_12197114.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12197114.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1249124d769e249c0b1746ca0368fc1a to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12197114.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12197114.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12197114.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12197114.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12197114.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12197114.xml
-- Model is restarting --
Adjusting last_iter, 378, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12197114.xml
last_cpu_time: 11390
upload_file_number: 15
last_iter: 359
last_upload: 1296000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0470_2011050100_123_980_12197114_0_r1819580094
The child process has been launched with process id: 2239593
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2239593:2239593] [20230115:114923:1673750963:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:52:48 STEP 360 H= 360:00 +CPU= 36.792
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:55:10 STEP 361 H= 361:00 +CPU= 20.989
11:55:31 STEP 362 H= 362:00 +CPU= 21.252
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:57:55 STEP 363 H= 363:00 +CPU= 32.658
11:58:16 STEP 364 H= 364:00 +CPU= 20.677
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:01:08 STEP 365 H= 365:00 +CPU= 20.562
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:05:22 STEP 366 H= 366:00 +CPU= 33.958
12:05:44 STEP 367 H= 367:00 +CPU= 21.404
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:09:16 STEP 368 H= 368:00 +CPU= 21.402
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:12:59 STEP 369 H= 369:00 +CPU= 32.467
12:13:21 STEP 370 H= 370:00 +CPU= 21.270
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:16:34 STEP 371 H= 371:00 +CPU= 21.783
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:19:53 STEP 372 H= 372:00 +CPU= 36.062
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000372
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:23:15 STEP 373 H= 373:00 +CPU= 20.713
12:23:37 STEP 374 H= 374:00 +CPU= 21.024
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:27:21 STEP 375 H= 375:00 +CPU= 32.444
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:30:24 STEP 376 H= 376:00 +CPU= 21.085
12:30:44 STEP 377 H= 377:00 +CPU= 20.450
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:34:20 STEP 378 H= 378:00 +CPU= 34.339
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:36:21 STEP 379 H= 379:00 +CPU= 20.150
12:36:42 STEP 380 H= 380:00 +CPU= 20.812
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:22:35 STEP 381 H= 381:00 +CPU= 32.076
22:22:56 STEP 382 H= 382:00 +CPU= 21.188
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:26:37 STEP 383 H= 383:00 +CPU= 20.508
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:29:27 STEP 384 H= 384:00 +CPU= 37.938
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000384
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000384
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000372
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0470_2011050100_123_980_12197114_0_r1819580094_15.zip
Uploading the intermediate file: upload_file_15.zip
22:29:45 STEP 385 H= 385:00 +CPU= 18.050
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:32:04 STEP 386 H= 386:00 +CPU= 19.368
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2011050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0470
(argv4) batchid: 980
(argv5) wuid: 12197114
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_800066ac9e16545a34d5ff2406f1f5a8 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0470_2011050100_123_980_12197114.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0470_2011050100_123_980_12197114.zip
ic_ancil_file: ic_ancil_12197114
ifsdata_file: ifsdata_12197114
climate_data_file: clim_data_12197114
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8c8abd1e077c18228d5f79e83b032356 to: /var/lib/boinc-client/slots/4/ic_ancil_12197114.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12197114.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1249124d769e249c0b1746ca0368fc1a to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12197114.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12197114.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12197114.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12197114.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12197114.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12197114.xml
-- Model is restarting --
Adjusting last_iter, 386, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12197114.xml
last_cpu_time: 12073
upload_file_number: 16
last_iter: 383
last_upload: 1382400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0470_2011050100_123_980_12197114_0_r1819580094
The child process has been launched with process id: 2360989
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360989:2360989] [20230115:223438:1673789678:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:37:18 STEP 384 H= 384:00 +CPU= 36.572
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:39:40 STEP 385 H= 385:00 +CPU= 20.983
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:41:33 STEP 386 H= 386:00 +CPU= 21.405
22:42:03 STEP 387 H= 387:00 +CPU= 30.778
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:44:46 STEP 388 H= 388:00 +CPU= 20.696
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:46:48 STEP 389 H= 389:00 +CPU= 20.253
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:49:02 STEP 390 H= 390:00 +CPU= 33.280
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:51:01 STEP 391 H= 391:00 +CPU= 19.263
22:51:22 STEP 392 H= 392:00 +CPU= 20.960
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:53:56 STEP 393 H= 393:00 +CPU= 31.680
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:56:08 STEP 394 H= 394:00 +CPU= 20.401
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:58:30 STEP 395 H= 395:00 +CPU= 20.834
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:00:47 STEP 396 H= 396:00 +CPU= 36.180
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000396
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:02:48 STEP 397 H= 397:00 +CPU= 20.159
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:05:00 STEP 398 H= 398:00 +CPU= 20.808
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:07:05 STEP 399 H= 399:00 +CPU= 32.446
23:07:26 STEP 400 H= 400:00 +CPU= 20.877
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:09:36 STEP 401 H= 401:00 +CPU= 20.233
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:13:23 STEP 402 H= 402:00 +CPU= 32.380
23:13:43 STEP 403 H= 403:00 +CPU= 20.255
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:15:36 STEP 404 H= 404:00 +CPU= 21.161
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:18:20 STEP 405 H= 405:00 +CPU= 32.099
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:20:22 STEP 406 H= 406:00 +CPU= 21.009
23:20:43 STEP 407 H= 407:00 +CPU= 21.498
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:23:33 STEP 408 H= 408:00 +CPU= 37.493
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000396
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0470_2011050100_123_980_12197114_0_r1819580094_16.zip
Uploading the intermediate file: upload_file_16.zip
23:23:51 STEP 409 H= 409:00 +CPU= 18.239
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:25:54 STEP 410 H= 410:00 +CPU= 19.709
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:28:39 STEP 411 H= 411:00 +CPU= 32.499
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:31:11 STEP 412 H= 412:00 +CPU= 20.757
23:31:31 STEP 413 H= 413:00 +CPU= 20.185
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:33:45 STEP 414 H= 414:00 +CPU= 33.013
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:36:06 STEP 415 H= 415:00 +CPU= 19.650
23:36:28 STEP 416 H= 416:00 +CPU= 21.348
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:40:43 STEP 417 H= 417:00 +CPU= 33.175
23:41:05 STEP 418 H= 418:00 +CPU= 21.701
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:43:17 STEP 419 H= 419:00 +CPU= 21.104
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:45:24 STEP 420 H= 420:00 +CPU= 35.447
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000420
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:48:06 STEP 421 H= 421:00 +CPU= 19.835
23:48:26 STEP 422 H= 422:00 +CPU= 20.771
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:50:20 STEP 423 H= 423:00 +CPU= 31.676
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:52:24 STEP 424 H= 424:00 +CPU= 21.304
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:54:15 STEP 425 H= 425:00 +CPU= 19.622
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:56:39 STEP 426 H= 426:00 +CPU= 32.933
23:56:59 STEP 427 H= 427:00 +CPU= 19.747
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:59:10 STEP 428 H= 428:00 +CPU= 19.779
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:01:43 STEP 429 H= 429:00 +CPU= 31.825
00:02:03 STEP 430 H= 430:00 +CPU= 20.148
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:04:55 STEP 431 H= 431:00 +CPU= 19.804
00:05:32 STEP 432 H= 432:00 +CPU= 36.545
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000420
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0470_2011050100_123_980_12197114_0_r1819580094_17.zip
Uploading the intermediate file: upload_file_17.zip
00:08:04 STEP 433 H= 433:00 +CPU= 19.847
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:10:46 STEP 434 H= 434:00 +CPU= 20.018
00:11:18 STEP 435 H= 435:00 +CPU= 31.987
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:13:51 STEP 436 H= 436:00 +CPU= 20.355
00:14:11 STEP 437 H= 437:00 +CPU= 20.018
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:16:44 STEP 438 H= 438:00 +CPU= 32.325
00:17:04 STEP 439 H= 439:00 +CPU= 20.103
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:19:36 STEP 440 H= 440:00 +CPU= 20.272
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:21:50 STEP 441 H= 441:00 +CPU= 31.962
00:22:10 STEP 442 H= 442:00 +CPU= 20.494
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:24:12 STEP 443 H= 443:00 +CPU= 19.838
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:26:58 STEP 444 H= 444:00 +CPU= 34.418
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000444
00:27:17 STEP 445 H= 445:00 +CPU= 19.390
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:29:08 STEP 446 H= 446:00 +CPU= 20.795
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:31:20 STEP 447 H= 447:00 +CPU= 31.243
00:31:40 STEP 448 H= 448:00 +CPU= 19.869
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:33:50 STEP 449 H= 449:00 +CPU= 19.252
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:35:54 STEP 450 H= 450:00 +CPU= 32.394
00:36:13 STEP 451 H= 451:00 +CPU= 19.452
00:36:34 STEP 452 H= 452:00 +CPU= 21.180
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:39:37 STEP 453 H= 453:00 +CPU= 31.863
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:41:59 STEP 454 H= 454:00 +CPU= 20.490
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:43:51 STEP 455 H= 455:00 +CPU= 20.162
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:46:20 STEP 456 H= 456:00 +CPU= 37.204
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMGGhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMSHhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114/ICMUAhq0f+000444
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0470_2011050100_123_980_12197114_0_r1819580094_18.zip
Uploading the intermediate file: upload_file_18.zip
00:46:39 STEP 457 H= 457:00 +CPU= 19.540
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:48:29 STEP 458 H= 458:00 +CPU= 19.073
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:50:52 STEP 459 H= 459:00 +CPU= 31.200
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:52:24 STEP 460 H= 460:00 +CPU= 19.900
00:52:44 STEP 461 H= 461:00 +CPU= 20.097
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2011050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0470
(argv4) batchid: 980
(argv5) wuid: 12197114
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197114
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_800066ac9e16545a34d5ff2406f1f5a8 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0470_2011050100_123_980_12197114.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0470_2011050100_123_980_12197114.zip
ic_ancil_file: ic_ancil_12197114
ifsdata_file: ifsdata_12197114
climate_data_file: clim_data_12197114
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8c8abd1e077c18228d5f79e83b032356 to: /var/lib/boinc-client/slots/4/ic_ancil_12197114.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12197114.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1249124d769e249c0b1746ca0368fc1a to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12197114.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12197114.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12197114.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12197114.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12197114.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12197114.xml
-- Model is restarting --
Adjusting last_iter, 461, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12197114.xml
last_cpu_time: 13992
upload_file_number: 19
last_iter: 455
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0470_2011050100_123_980_12197114_0_r1819580094
The child process has been launched with process id: 2380676
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2380676:2380676] [20230116:005512:1673798112:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
00:55:50 STEP 456 H= 456:00 +CPU= 35.199
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:58:21 STEP 457 H= 457:00 +CPU= 19.663
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
01:00:22 STEP 458 H= 458:00 +CPU= 20.425
01:00:54 STEP 459 H= 459:00 +CPU= 31.446
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
01:03:36 STEP 460 H= 460:00 +CPU= 20.160
01:03:55 STEP 461 H= 461:00 +CPU= 19.299
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
01:06:29 STEP 462 H= 462:00 +CPU= 32.352
01:06:48 STEP 463 H= 463:00 +CPU= 19.361
01:07:09 STEP 464 H= 464:00 +CPU= 20.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
01:10:21 STEP 465 H= 465:00 +CPU= 32.242
01:10:41 STEP 466 H= 466:00 +CPU= 19.604
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
01:13:12 STEP 467 H= 467:00 +CPU= 19.743
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
01:14:58 STEP 468 H= 468:00 +CPU= 35.653
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000468
01:15:18 STEP 469 H= 469:00 +CPU= 19.344
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
01:18:20 STEP 470 H= 470:00 +CPU= 20.583
01:18:51 STEP 471 H= 471:00 +CPU= 31.711
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
01:21:24 STEP 472 H= 472:00 +CPU= 20.772
01:21:44 STEP 473 H= 473:00 +CPU= 19.691
01:22:16 STEP 474 H= 474:00 +CPU= 32.247
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
01:24:57 STEP 475 H= 475:00 +CPU= 19.103
01:25:17 STEP 476 H= 476:00 +CPU= 20.401
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
01:27:50 STEP 477 H= 477:00 +CPU= 31.648
01:28:10 STEP 478 H= 478:00 +CPU= 19.784
01:28:30 STEP 479 H= 479:00 +CPU= 20.695
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
01:31:57 STEP 480 H= 480:00 +CPU= 36.691
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
