climateprediction.net home page
Task 22259412

Task 22259412

Name oifs_43r3_ps_0096_1989050100_123_958_12174740_0
Workunit 12174740
Created 21 Dec 2022, 15:39:23 UTC
Sent 21 Dec 2022, 15:42:10 UTC
Report deadline 20 Jan 2023, 15:42:10 UTC
Received 22 Dec 2022, 11:17:29 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1519251
Run time 5 hours 57 min 46 sec
CPU time 5 hours 50 min 6 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.97 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,410.86 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,674.17 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 the BOINC client, suspending the child process
Resuming the child process
  11:11:11 STEP  504 H= 504:00 +CPU= 42.145
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:13:53 STEP  505 H= 505:00 +CPU= 20.995
  11:14:17 STEP  506 H= 506:00 +CPU= 23.595
  11:14:51 STEP  507 H= 507:00 +CPU= 33.365
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:22:56 STEP  508 H= 508:00 +CPU= 21.833
  11:23:21 STEP  509 H= 509:00 +CPU= 24.209
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:27:11 STEP  510 H= 510:00 +CPU= 38.089
  11:27:36 STEP  511 H= 511:00 +CPU= 24.058
  11:28:00 STEP  512 H= 512:00 +CPU= 23.374
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:45:00 STEP  513 H= 513:00 +CPU= 37.463
  11:45:25 STEP  514 H= 514:00 +CPU= 24.377
  11:45:49 STEP  515 H= 515:00 +CPU= 24.165
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:47:48 STEP  516 H= 516:00 +CPU= 38.237
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000516
  11:48:09 STEP  517 H= 517:00 +CPU= 20.493
  11:48:30 STEP  518 H= 518:00 +CPU= 21.151
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:50:15 STEP  519 H= 519:00 +CPU= 32.775
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:51:46 STEP  520 H= 520:00 +CPU= 21.010
  11:52:08 STEP  521 H= 521:00 +CPU= 21.284
  11:52:40 STEP  522 H= 522:00 +CPU= 32.683
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:54:33 STEP  523 H= 523:00 +CPU= 20.449
  11:54:55 STEP  524 H= 524:00 +CPU= 21.525
  11:55:27 STEP  525 H= 525:00 +CPU= 32.081
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:56:39 STEP  526 H= 526:00 +CPU= 21.366
  11:57:00 STEP  527 H= 527:00 +CPU= 20.729
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:59:00 STEP  528 H= 528:00 +CPU= 38.355
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000516
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_21.zip
Uploading the intermediate file: upload_file_21.zip
  11:59:20 STEP  529 H= 529:00 +CPU= 20.084
  11:59:41 STEP  530 H= 530:00 +CPU= 21.184
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:01:24 STEP  531 H= 531:00 +CPU= 32.531
  12:01:45 STEP  532 H= 532:00 +CPU= 21.007
  12:02:05 STEP  533 H= 533:00 +CPU= 19.892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:03:50 STEP  534 H= 534:00 +CPU= 33.368
  12:04:10 STEP  535 H= 535:00 +CPU= 20.325
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:05:43 STEP  536 H= 536:00 +CPU= 21.822
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:07:17 STEP  537 H= 537:00 +CPU= 33.070
  12:07:38 STEP  538 H= 538:00 +CPU= 21.227
  12:07:58 STEP  539 H= 539:00 +CPU= 20.311
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:09:36 STEP  540 H= 540:00 +CPU= 36.373
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000540
  12:09:56 STEP  541 H= 541:00 +CPU= 20.229
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:11:40 STEP  542 H= 542:00 +CPU= 21.334
  12:12:13 STEP  543 H= 543:00 +CPU= 33.088
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:13:35 STEP  544 H= 544:00 +CPU= 21.061
  12:13:56 STEP  545 H= 545:00 +CPU= 20.965
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:15:41 STEP  546 H= 546:00 +CPU= 33.339
  12:16:02 STEP  547 H= 547:00 +CPU= 21.009
  12:16:23 STEP  548 H= 548:00 +CPU= 20.789
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:18:16 STEP  549 H= 549:00 +CPU= 31.885
  12:18:38 STEP  550 H= 550:00 +CPU= 21.834
  12:18:59 STEP  551 H= 551:00 +CPU= 20.591
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:21:19 STEP  552 H= 552:00 +CPU= 38.485
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000540
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_22.zip
Uploading the intermediate file: upload_file_22.zip
  12:21:39 STEP  553 H= 553:00 +CPU= 20.076
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:23:38 STEP  554 H= 554:00 +CPU= 21.134
  12:24:11 STEP  555 H= 555:00 +CPU= 33.311
  12:24:32 STEP  556 H= 556:00 +CPU= 20.932
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:26:23 STEP  557 H= 557:00 +CPU= 20.334
  12:26:56 STEP  558 H= 558:00 +CPU= 33.429
  12:27:20 STEP  559 H= 559:00 +CPU= 23.173
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:33:07 STEP  560 H= 560:00 +CPU= 24.333
  12:33:45 STEP  561 H= 561:00 +CPU= 37.185
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:52:12 STEP  562 H= 562:00 +CPU= 23.314
  12:52:35 STEP  563 H= 563:00 +CPU= 22.504
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:00:21 STEP  564 H= 564:00 +CPU= 42.707
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000564
  13:00:45 STEP  565 H= 565:00 +CPU= 23.249
  13:01:09 STEP  566 H= 566:00 +CPU= 23.722
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0096
(argv4) batchid: 958
(argv5) wuid: 12174740
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_02bb1386e77f1f6a5098ca4ee71ac874 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip
ic_ancil_file: ic_ancil_12174740
ifsdata_file: ifsdata_12174740
climate_data_file: clim_data_12174740
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3d423343c5e75ef253b31f596e3dabad to: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_add8a1baf742cd15861b692d5491858d to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12174740.xml
-- Model is restarting --
Adjusting last_iter, 567, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml
last_cpu_time: 16369
upload_file_number: 23
last_iter: 551
last_upload: 1987200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001
The child process has been launched with process id: 3234650
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:25:27 STEP  552 H= 552:00 +CPU= 40.488
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:34:31 STEP  553 H= 553:00 +CPU= 21.627
  13:34:52 STEP  554 H= 554:00 +CPU= 21.382
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:36:58 STEP  555 H= 555:00 +CPU= 33.393
  13:37:19 STEP  556 H= 556:00 +CPU= 21.113
  13:37:39 STEP  557 H= 557:00 +CPU= 20.089
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:39:34 STEP  558 H= 558:00 +CPU= 33.522
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:41:07 STEP  559 H= 559:00 +CPU= 21.221
  13:41:28 STEP  560 H= 560:00 +CPU= 20.923
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:43:22 STEP  561 H= 561:00 +CPU= 32.714
  13:43:43 STEP  562 H= 562:00 +CPU= 20.593
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:45:17 STEP  563 H= 563:00 +CPU= 21.903
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:48:15 STEP  564 H= 564:00 +CPU= 35.747
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000564
  13:48:37 STEP  565 H= 565:00 +CPU= 21.402
  13:48:58 STEP  566 H= 566:00 +CPU= 20.648
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:50:53 STEP  567 H= 567:00 +CPU= 33.179
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:52:17 STEP  568 H= 568:00 +CPU= 21.728
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:53:31 STEP  569 H= 569:00 +CPU= 21.798
  13:54:03 STEP  570 H= 570:00 +CPU= 32.041
  13:54:24 STEP  571 H= 571:00 +CPU= 20.487
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:56:07 STEP  572 H= 572:00 +CPU= 21.543
  13:56:39 STEP  573 H= 573:00 +CPU= 32.134
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:58:42 STEP  574 H= 574:00 +CPU= 21.596
  13:59:04 STEP  575 H= 575:00 +CPU= 21.898
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:00:54 STEP  576 H= 576:00 +CPU= 38.636
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000564
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_23.zip
Uploading the intermediate file: upload_file_23.zip
  14:01:14 STEP  577 H= 577:00 +CPU= 20.298
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:03:01 STEP  578 H= 578:00 +CPU= 20.880
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:04:37 STEP  579 H= 579:00 +CPU= 33.417
  14:04:58 STEP  580 H= 580:00 +CPU= 21.060
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:06:50 STEP  581 H= 581:00 +CPU= 20.575
  14:07:24 STEP  582 H= 582:00 +CPU= 34.174
  14:07:44 STEP  583 H= 583:00 +CPU= 19.928
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:10:17 STEP  584 H= 584:00 +CPU= 21.490
  14:10:49 STEP  585 H= 585:00 +CPU= 32.167
  14:11:11 STEP  586 H= 586:00 +CPU= 21.048
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:12:54 STEP  587 H= 587:00 +CPU= 21.436
  14:13:30 STEP  588 H= 588:00 +CPU= 35.714
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000588
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:15:22 STEP  589 H= 589:00 +CPU= 20.856
  14:15:44 STEP  590 H= 590:00 +CPU= 21.431
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:17:47 STEP  591 H= 591:00 +CPU= 32.771
  14:18:08 STEP  592 H= 592:00 +CPU= 21.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:19:49 STEP  593 H= 593:00 +CPU= 19.974
  14:20:23 STEP  594 H= 594:00 +CPU= 33.334
  14:20:43 STEP  595 H= 595:00 +CPU= 20.138
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:22:37 STEP  596 H= 596:00 +CPU= 22.616
  14:23:10 STEP  597 H= 597:00 +CPU= 32.189
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:25:23 STEP  598 H= 598:00 +CPU= 20.856
  14:25:44 STEP  599 H= 599:00 +CPU= 21.602
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:27:44 STEP  600 H= 600:00 +CPU= 39.406
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_24.zip
Uploading the intermediate file: upload_file_24.zip
  14:28:05 STEP  601 H= 601:00 +CPU= 20.204
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:30:06 STEP  602 H= 602:00 +CPU= 21.618
  14:30:38 STEP  603 H= 603:00 +CPU= 32.545
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:32:51 STEP  604 H= 604:00 +CPU= 21.057
  14:33:12 STEP  605 H= 605:00 +CPU= 21.396
  14:33:46 STEP  606 H= 606:00 +CPU= 33.233
  14:34:07 STEP  607 H= 607:00 +CPU= 20.808
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:35:49 STEP  608 H= 608:00 +CPU= 21.471
  14:36:23 STEP  609 H= 609:00 +CPU= 32.977
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:37:57 STEP  610 H= 610:00 +CPU= 22.360
  14:38:18 STEP  611 H= 611:00 +CPU= 20.543
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:40:05 STEP  612 H= 612:00 +CPU= 36.205
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000612
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:42:18 STEP  613 H= 613:00 +CPU= 20.936
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:44:41 STEP  614 H= 614:00 +CPU= 21.524
  14:45:15 STEP  615 H= 615:00 +CPU= 33.131
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:48:07 STEP  616 H= 616:00 +CPU= 21.543
  14:48:29 STEP  617 H= 617:00 +CPU= 22.651
  14:49:03 STEP  618 H= 618:00 +CPU= 33.531
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:50:45 STEP  619 H= 619:00 +CPU= 20.773
  14:51:07 STEP  620 H= 620:00 +CPU= 21.432
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:53:02 STEP  621 H= 621:00 +CPU= 33.721
  14:53:23 STEP  622 H= 622:00 +CPU= 20.853
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:55:05 STEP  623 H= 623:00 +CPU= 21.259
  14:55:43 STEP  624 H= 624:00 +CPU= 37.760
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_25.zip
Uploading the intermediate file: upload_file_25.zip
  14:57:56 STEP  625 H= 625:00 +CPU= 20.847
  14:58:18 STEP  626 H= 626:00 +CPU= 22.122
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:00:12 STEP  627 H= 627:00 +CPU= 33.127
  15:00:35 STEP  628 H= 628:00 +CPU= 22.054
  15:00:55 STEP  629 H= 629:00 +CPU= 20.635
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:03:41 STEP  630 H= 630:00 +CPU= 33.815
  15:04:02 STEP  631 H= 631:00 +CPU= 20.100
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:05:23 STEP  632 H= 632:00 +CPU= 21.428
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:06:37 STEP  633 H= 633:00 +CPU= 32.371
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:08:20 STEP  634 H= 634:00 +CPU= 20.833
  15:08:41 STEP  635 H= 635:00 +CPU= 21.015
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:10:37 STEP  636 H= 636:00 +CPU= 35.595
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000636
  15:10:58 STEP  637 H= 637:00 +CPU= 21.630
  15:11:20 STEP  638 H= 638:00 +CPU= 21.069
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:13:14 STEP  639 H= 639:00 +CPU= 32.415
  15:13:35 STEP  640 H= 640:00 +CPU= 20.746
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:15:27 STEP  641 H= 641:00 +CPU= 20.667
  15:16:00 STEP  642 H= 642:00 +CPU= 33.343
  15:16:21 STEP  643 H= 643:00 +CPU= 20.086
  15:16:42 STEP  644 H= 644:00 +CPU= 21.335
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:18:36 STEP  645 H= 645:00 +CPU= 32.549
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:20:18 STEP  646 H= 646:00 +CPU= 20.516
  15:20:40 STEP  647 H= 647:00 +CPU= 22.221
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0096
(argv4) batchid: 958
(argv5) wuid: 12174740
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_02bb1386e77f1f6a5098ca4ee71ac874 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip
ic_ancil_file: ic_ancil_12174740
ifsdata_file: ifsdata_12174740
climate_data_file: clim_data_12174740
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3d423343c5e75ef253b31f596e3dabad to: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_add8a1baf742cd15861b692d5491858d to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12174740.xml
-- Model is restarting --
Adjusting last_iter, 647, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml
last_cpu_time: 18812
upload_file_number: 26
last_iter: 623
last_upload: 2246400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001
The child process has been launched with process id: 3254922
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:25:19 STEP  648 H= 648:00 +CPU= 37.406
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_26.zip
Uploading the intermediate file: upload_file_26.zip
  15:25:39 STEP  649 H= 649:00 +CPU= 20.007
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:27:15 STEP  650 H= 650:00 +CPU= 19.907
  15:27:48 STEP  651 H= 651:00 +CPU= 32.820
  15:28:09 STEP  652 H= 652:00 +CPU= 20.932
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:30:00 STEP  653 H= 653:00 +CPU= 20.299
  15:30:33 STEP  654 H= 654:00 +CPU= 32.725
  15:30:53 STEP  655 H= 655:00 +CPU= 19.956
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:32:56 STEP  656 H= 656:00 +CPU= 21.295
  15:33:28 STEP  657 H= 657:00 +CPU= 31.927
  15:33:49 STEP  658 H= 658:00 +CPU= 20.880
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:35:42 STEP  659 H= 659:00 +CPU= 20.494
  15:36:18 STEP  660 H= 660:00 +CPU= 35.787
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000660
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:38:09 STEP  661 H= 661:00 +CPU= 20.634
  15:38:31 STEP  662 H= 662:00 +CPU= 21.613
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:40:24 STEP  663 H= 663:00 +CPU= 32.185
  15:40:45 STEP  664 H= 664:00 +CPU= 21.617
  15:41:06 STEP  665 H= 665:00 +CPU= 20.202
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:43:30 STEP  666 H= 666:00 +CPU= 33.043
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:44:53 STEP  667 H= 667:00 +CPU= 20.598
  15:45:14 STEP  668 H= 668:00 +CPU= 21.526
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:46:49 STEP  669 H= 669:00 +CPU= 33.305
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:48:30 STEP  670 H= 670:00 +CPU= 20.946
  15:48:52 STEP  671 H= 671:00 +CPU= 21.472
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:50:52 STEP  672 H= 672:00 +CPU= 38.623
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_27.zip
Uploading the intermediate file: upload_file_27.zip
  15:51:12 STEP  673 H= 673:00 +CPU= 20.481
  15:51:33 STEP  674 H= 674:00 +CPU= 20.865
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:53:28 STEP  675 H= 675:00 +CPU= 33.415
  15:53:49 STEP  676 H= 676:00 +CPU= 20.899
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:01:22 STEP  677 H= 677:00 +CPU= 21.898
  16:01:58 STEP  678 H= 678:00 +CPU= 35.196
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:12:52 STEP  679 H= 679:00 +CPU= 22.397
  16:13:15 STEP  680 H= 680:00 +CPU= 22.073
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:17:41 STEP  681 H= 681:00 +CPU= 35.150
  16:18:03 STEP  682 H= 682:00 +CPU= 21.762
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:23:47 STEP  683 H= 683:00 +CPU= 22.648
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:29:09 STEP  684 H= 684:00 +CPU= 39.599
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000684
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:32 STEP  685 H= 685:00 +CPU= 21.702
  16:36:55 STEP  686 H= 686:00 +CPU= 22.428
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:43:01 STEP  687 H= 687:00 +CPU= 34.350
  16:43:24 STEP  688 H= 688:00 +CPU= 23.118
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:47:27 STEP  689 H= 689:00 +CPU= 21.678
  16:48:03 STEP  690 H= 690:00 +CPU= 35.242
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:51:04 STEP  691 H= 691:00 +CPU= 21.067
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:54:58 STEP  692 H= 692:00 +CPU= 22.612
  16:55:33 STEP  693 H= 693:00 +CPU= 34.400
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:00:38 STEP  694 H= 694:00 +CPU= 22.767
  17:01:02 STEP  695 H= 695:00 +CPU= 23.644
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:12:37 STEP  696 H= 696:00 +CPU= 40.584
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_28.zip
Uploading the intermediate file: upload_file_28.zip
  17:13:00 STEP  697 H= 697:00 +CPU= 22.340
  17:13:22 STEP  698 H= 698:00 +CPU= 22.492
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:17:10 STEP  699 H= 699:00 +CPU= 35.083
  17:17:33 STEP  700 H= 700:00 +CPU= 22.942
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:23:37 STEP  701 H= 701:00 +CPU= 21.802
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:29:45 STEP  702 H= 702:00 +CPU= 35.847
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0096
(argv4) batchid: 958
(argv5) wuid: 12174740
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_02bb1386e77f1f6a5098ca4ee71ac874 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip
ic_ancil_file: ic_ancil_12174740
ifsdata_file: ifsdata_12174740
climate_data_file: clim_data_12174740
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3d423343c5e75ef253b31f596e3dabad to: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_add8a1baf742cd15861b692d5491858d to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12174740.xml
-- Model is restarting --
Adjusting last_iter, 703, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml
last_cpu_time: 20263
upload_file_number: 29
last_iter: 695
last_upload: 2505600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001
The child process has been launched with process id: 3274993
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:40:32 STEP  696 H= 696:00 +CPU= 39.198
  17:40:55 STEP  697 H= 697:00 +CPU= 22.405
  17:41:18 STEP  698 H= 698:00 +CPU= 23.187
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:47:36 STEP  699 H= 699:00 +CPU= 35.298
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:51:32 STEP  700 H= 700:00 +CPU= 23.295
  17:51:54 STEP  701 H= 701:00 +CPU= 21.725
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:56:23 STEP  702 H= 702:00 +CPU= 36.151
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:59:07 STEP  703 H= 703:00 +CPU= 22.427
  17:59:30 STEP  704 H= 704:00 +CPU= 22.809
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:03:06 STEP  705 H= 705:00 +CPU= 35.096
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:07:32 STEP  706 H= 706:00 +CPU= 23.068
  18:07:55 STEP  707 H= 707:00 +CPU= 23.200
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:11:37 STEP  708 H= 708:00 +CPU= 38.589
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000708
  18:12:00 STEP  709 H= 709:00 +CPU= 22.711
  18:12:22 STEP  710 H= 710:00 +CPU= 22.103
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:18:29 STEP  711 H= 711:00 +CPU= 35.360
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:21:53 STEP  712 H= 712:00 +CPU= 22.709
  18:22:16 STEP  713 H= 713:00 +CPU= 22.382
  18:22:51 STEP  714 H= 714:00 +CPU= 34.855
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:25:55 STEP  715 H= 715:00 +CPU= 22.877
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:30:00 STEP  716 H= 716:00 +CPU= 21.966
  18:30:37 STEP  717 H= 717:00 +CPU= 36.041
  18:30:59 STEP  718 H= 718:00 +CPU= 22.213
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:37:14 STEP  719 H= 719:00 +CPU= 22.837
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:45:08 STEP  720 H= 720:00 +CPU= 40.384
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_29.zip
Uploading the intermediate file: upload_file_29.zip
  18:45:30 STEP  721 H= 721:00 +CPU= 21.982
  18:45:53 STEP  722 H= 722:00 +CPU= 22.606
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net