climateprediction.net home page
Task 22299192

Task 22299192

Name oifs_43r3_ps_0070_1993050100_123_962_12178714_2
Workunit 12178714
Created 17 Jan 2023, 22:35:10 UTC
Sent 17 Jan 2023, 22:41:12 UTC
Report deadline 18 Mar 2023, 22:41:12 UTC
Received 20 Jan 2023, 18:32:21 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1519251
Run time 7 hours 8 min 28 sec
CPU time 5 hours 55 min 40 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.94 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.32 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,676.30 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
rgv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/16
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_56435e71a077fe9b66cece1d0898629b to: /var/lib/boinc-client/slots/16/oifs_43r3_ps_0070_1993050100_123_962_12178714.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/16/oifs_43r3_ps_0070_1993050100_123_962_12178714.zip
ic_ancil_file: ic_ancil_12178714
ifsdata_file: ifsdata_12178714
climate_data_file: clim_data_12178714
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5aa645fb931939c92f4159ccf1c5f718 to: /var/lib/boinc-client/slots/16/ic_ancil_12178714.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/16/ic_ancil_12178714.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4509268e533680c53f92685772cbe318 to: /var/lib/boinc-client/slots/16/ifsdata/ifsdata_12178714.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/16/ifsdata/ifsdata_12178714.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/16/159l_2/clim_data_12178714.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/16/159l_2/clim_data_12178714.zip
Checking for progress XML file: /var/lib/boinc-client/slots/16/progress_file_12178714.xml
Opened progress file ok : /var/lib/boinc-client/slots/16/progress_file_12178714.xml
-- Model is restarting --
Adjusting last_iter, 568, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/16/progress_file_12178714.xml
last_cpu_time: 15500
upload_file_number: 23
last_iter: 551
last_upload: 1987200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204
The child process has been launched with process id: 332624
Executing the command: /var/lib/boinc-client/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:332624:332624] [20230120:221420:1674220460:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:18:08 STEP  552 H= 552:00 +CPU= 41.628
  22:18:32 STEP  553 H= 553:00 +CPU= 23.122
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:23:37 STEP  554 H= 554:00 +CPU= 24.002
  22:24:15 STEP  555 H= 555:00 +CPU= 36.639
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:28:51 STEP  556 H= 556:00 +CPU= 23.361
  22:29:14 STEP  557 H= 557:00 +CPU= 23.103
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:35:05 STEP  558 H= 558:00 +CPU= 38.116
  22:35:28 STEP  559 H= 559:00 +CPU= 22.748
  22:35:52 STEP  560 H= 560:00 +CPU= 23.533
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:51:04 STEP  561 H= 561:00 +CPU= 36.915
  22:51:25 STEP  562 H= 562:00 +CPU= 20.928
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:53:08 STEP  563 H= 563:00 +CPU= 21.152
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:55:04 STEP  564 H= 564:00 +CPU= 35.423
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000564
  22:55:25 STEP  565 H= 565:00 +CPU= 20.737
  22:55:46 STEP  566 H= 566:00 +CPU= 20.348
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:57:49 STEP  567 H= 567:00 +CPU= 31.809
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:59:21 STEP  568 H= 568:00 +CPU= 20.440
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:00:43 STEP  569 H= 569:00 +CPU= 19.998
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:02:07 STEP  570 H= 570:00 +CPU= 32.614
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:03:27 STEP  571 H= 571:00 +CPU= 20.179
  23:03:47 STEP  572 H= 572:00 +CPU= 20.241
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:05:10 STEP  573 H= 573:00 +CPU= 31.562
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:06:43 STEP  574 H= 574:00 +CPU= 21.342
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:08:05 STEP  575 H= 575:00 +CPU= 20.663
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:09:55 STEP  576 H= 576:00 +CPU= 37.941
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000564
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204_23.zip
Uploading the intermediate file: upload_file_23.zip
  23:10:15 STEP  577 H= 577:00 +CPU= 19.593
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:11:37 STEP  578 H= 578:00 +CPU= 20.748
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:13:09 STEP  579 H= 579:00 +CPU= 31.182
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:14:41 STEP  580 H= 580:00 +CPU= 20.533
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:16:34 STEP  581 H= 581:00 +CPU= 20.672
  23:17:06 STEP  582 H= 582:00 +CPU= 32.242
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:18:36 STEP  583 H= 583:00 +CPU= 19.554
  23:18:57 STEP  584 H= 584:00 +CPU= 21.365
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:20:39 STEP  585 H= 585:00 +CPU= 31.877
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:22:00 STEP  586 H= 586:00 +CPU= 20.785
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:23:22 STEP  587 H= 587:00 +CPU= 20.516
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:25:18 STEP  588 H= 588:00 +CPU= 34.754
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000588
  23:25:38 STEP  589 H= 589:00 +CPU= 20.394
  23:25:59 STEP  590 H= 590:00 +CPU= 20.540
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:27:51 STEP  591 H= 591:00 +CPU= 31.888
  23:28:11 STEP  592 H= 592:00 +CPU= 20.600
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:30:03 STEP  593 H= 593:00 +CPU= 20.329
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:31:37 STEP  594 H= 594:00 +CPU= 33.258
  23:31:57 STEP  595 H= 595:00 +CPU= 19.996
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:33:30 STEP  596 H= 596:00 +CPU= 20.437
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:35:13 STEP  597 H= 597:00 +CPU= 32.094
  23:35:34 STEP  598 H= 598:00 +CPU= 21.124
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:37:16 STEP  599 H= 599:00 +CPU= 20.873
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:38:45 STEP  600 H= 600:00 +CPU= 38.488
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204_24.zip
Uploading the intermediate file: upload_file_24.zip
  23:39:05 STEP  601 H= 601:00 +CPU= 19.908
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:41:06 STEP  602 H= 602:00 +CPU= 20.615
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:43:00 STEP  603 H= 603:00 +CPU= 31.951
  23:43:21 STEP  604 H= 604:00 +CPU= 20.958
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:44:52 STEP  605 H= 605:00 +CPU= 19.875
  23:45:24 STEP  606 H= 606:00 +CPU= 32.230
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:47:06 STEP  607 H= 607:00 +CPU= 21.032
  23:47:26 STEP  608 H= 608:00 +CPU= 20.230
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:49:19 STEP  609 H= 609:00 +CPU= 31.585
  23:49:40 STEP  610 H= 610:00 +CPU= 21.050
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:51:11 STEP  611 H= 611:00 +CPU= 19.951
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:52:48 STEP  612 H= 612:00 +CPU= 35.050
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000612
  23:53:08 STEP  613 H= 613:00 +CPU= 19.981
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:54:51 STEP  614 H= 614:00 +CPU= 21.136
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:56:35 STEP  615 H= 615:00 +CPU= 31.866
  23:56:56 STEP  616 H= 616:00 +CPU= 21.029
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:58:27 STEP  617 H= 617:00 +CPU= 19.879
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:59:59 STEP  618 H= 618:00 +CPU= 31.916
  00:00:20 STEP  619 H= 619:00 +CPU= 20.768
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:01:51 STEP  620 H= 620:00 +CPU= 20.821
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:03:14 STEP  621 H= 621:00 +CPU= 32.627
  00:03:35 STEP  622 H= 622:00 +CPU= 20.299
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:05:08 STEP  623 H= 623:00 +CPU= 21.255
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:06:57 STEP  624 H= 624:00 +CPU= 37.453
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204_25.zip
Uploading the intermediate file: upload_file_25.zip
  00:07:17 STEP  625 H= 625:00 +CPU= 19.894
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:09:14 STEP  626 H= 626:00 +CPU= 20.655
  00:09:46 STEP  627 H= 627:00 +CPU= 31.967
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:11:39 STEP  628 H= 628:00 +CPU= 21.135
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:13:31 STEP  629 H= 629:00 +CPU= 19.953
  00:14:04 STEP  630 H= 630:00 +CPU= 33.248
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:15:35 STEP  631 H= 631:00 +CPU= 20.488
  00:15:54 STEP  632 H= 632:00 +CPU= 19.740
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0070
(argv4) batchid: 962
(argv5) wuid: 12178714
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/16
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_56435e71a077fe9b66cece1d0898629b to: /var/lib/boinc-client/slots/16/oifs_43r3_ps_0070_1993050100_123_962_12178714.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/16/oifs_43r3_ps_0070_1993050100_123_962_12178714.zip
ic_ancil_file: ic_ancil_12178714
ifsdata_file: ifsdata_12178714
climate_data_file: clim_data_12178714
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5aa645fb931939c92f4159ccf1c5f718 to: /var/lib/boinc-client/slots/16/ic_ancil_12178714.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/16/ic_ancil_12178714.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4509268e533680c53f92685772cbe318 to: /var/lib/boinc-client/slots/16/ifsdata/ifsdata_12178714.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/16/ifsdata/ifsdata_12178714.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/16/159l_2/clim_data_12178714.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/16/159l_2/clim_data_12178714.zip
Checking for progress XML file: /var/lib/boinc-client/slots/16/progress_file_12178714.xml
Opened progress file ok : /var/lib/boinc-client/slots/16/progress_file_12178714.xml
-- Model is restarting --
Adjusting last_iter, 633, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/16/progress_file_12178714.xml
last_cpu_time: 17552
upload_file_number: 26
last_iter: 623
last_upload: 2246400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204
The child process has been launched with process id: 350552
Executing the command: /var/lib/boinc-client/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:350552:350552] [20230121:001732:1674227852:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:19:32 STEP  624 H= 624:00 +CPU= 36.313
  00:19:53 STEP  625 H= 625:00 +CPU= 20.569
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:21:14 STEP  626 H= 626:00 +CPU= 20.592
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:22:28 STEP  627 H= 627:00 +CPU= 32.281
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:23:40 STEP  628 H= 628:00 +CPU= 21.296
  00:24:00 STEP  629 H= 629:00 +CPU= 19.904
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:25:53 STEP  630 H= 630:00 +CPU= 32.332
  00:26:12 STEP  631 H= 631:00 +CPU= 19.716
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:27:53 STEP  632 H= 632:00 +CPU= 20.765
  00:28:25 STEP  633 H= 633:00 +CPU= 31.925
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:30:28 STEP  634 H= 634:00 +CPU= 21.019
  00:30:49 STEP  635 H= 635:00 +CPU= 20.068
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:32:54 STEP  636 H= 636:00 +CPU= 34.730
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000636
  00:33:15 STEP  637 H= 637:00 +CPU= 20.364
  00:33:36 STEP  638 H= 638:00 +CPU= 20.900
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:35:18 STEP  639 H= 639:00 +CPU= 31.163
  00:35:38 STEP  640 H= 640:00 +CPU= 20.156
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:37:10 STEP  641 H= 641:00 +CPU= 20.883
  00:37:42 STEP  642 H= 642:00 +CPU= 32.001
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:39:34 STEP  643 H= 643:00 +CPU= 20.047
  00:39:54 STEP  644 H= 644:00 +CPU= 20.577
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:41:35 STEP  645 H= 645:00 +CPU= 31.206
  00:41:57 STEP  646 H= 646:00 +CPU= 21.592
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:43:29 STEP  647 H= 647:00 +CPU= 20.941
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:45:28 STEP  648 H= 648:00 +CPU= 37.813
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204_26.zip
Uploading the intermediate file: upload_file_26.zip
  00:45:48 STEP  649 H= 649:00 +CPU= 19.551
  00:46:09 STEP  650 H= 650:00 +CPU= 20.796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:48:55 STEP  651 H= 651:00 +CPU= 32.164
  00:49:16 STEP  652 H= 652:00 +CPU= 21.414
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:50:48 STEP  653 H= 653:00 +CPU= 20.664
  00:51:21 STEP  654 H= 654:00 +CPU= 32.359
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:53:02 STEP  655 H= 655:00 +CPU= 19.984
  00:53:23 STEP  656 H= 656:00 +CPU= 20.792
  00:53:54 STEP  657 H= 657:00 +CPU= 31.530
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:55:57 STEP  658 H= 658:00 +CPU= 20.813
  00:56:17 STEP  659 H= 659:00 +CPU= 20.121
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:58:24 STEP  660 H= 660:00 +CPU= 35.283
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000660
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:59:35 STEP  661 H= 661:00 +CPU= 20.609
  00:59:55 STEP  662 H= 662:00 +CPU= 20.013
  01:00:27 STEP  663 H= 663:00 +CPU= 31.124
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:02:09 STEP  664 H= 664:00 +CPU= 20.297
  01:02:29 STEP  665 H= 665:00 +CPU= 20.352
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:04:04 STEP  666 H= 666:00 +CPU= 32.429
  01:04:24 STEP  667 H= 667:00 +CPU= 19.945
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:05:57 STEP  668 H= 668:00 +CPU= 20.462
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:07:31 STEP  669 H= 669:00 +CPU= 31.983
  01:07:52 STEP  670 H= 670:00 +CPU= 20.845
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:09:04 STEP  671 H= 671:00 +CPU= 20.457
  01:09:41 STEP  672 H= 672:00 +CPU= 37.332
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204_27.zip
Uploading the intermediate file: upload_file_27.zip
  01:11:22 STEP  673 H= 673:00 +CPU= 19.568
  01:11:43 STEP  674 H= 674:00 +CPU= 20.634
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:13:26 STEP  675 H= 675:00 +CPU= 32.009
  01:13:47 STEP  676 H= 676:00 +CPU= 21.010
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:15:27 STEP  677 H= 677:00 +CPU= 19.363
  01:16:00 STEP  678 H= 678:00 +CPU= 31.953
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:17:30 STEP  679 H= 679:00 +CPU= 19.450
  01:17:50 STEP  680 H= 680:00 +CPU= 20.191
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:19:33 STEP  681 H= 681:00 +CPU= 31.691
  01:19:53 STEP  682 H= 682:00 +CPU= 19.979
  01:20:13 STEP  683 H= 683:00 +CPU= 19.358
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:22:11 STEP  684 H= 684:00 +CPU= 35.916
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000684
  01:22:30 STEP  685 H= 685:00 +CPU= 19.391
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:24:02 STEP  686 H= 686:00 +CPU= 20.332
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:25:47 STEP  687 H= 687:00 +CPU= 32.343
  01:26:07 STEP  688 H= 688:00 +CPU= 20.035
  01:26:27 STEP  689 H= 689:00 +CPU= 20.230
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:28:31 STEP  690 H= 690:00 +CPU= 32.174
  01:28:50 STEP  691 H= 691:00 +CPU= 19.566
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:30:12 STEP  692 H= 692:00 +CPU= 20.527
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:31:45 STEP  693 H= 693:00 +CPU= 31.710
  01:32:05 STEP  694 H= 694:00 +CPU= 20.524
  01:32:26 STEP  695 H= 695:00 +CPU= 20.980
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:34:15 STEP  696 H= 696:00 +CPU= 37.325
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204_28.zip
Uploading the intermediate file: upload_file_28.zip
  01:34:32 STEP  697 H= 697:00 +CPU= 16.981
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:35:42 STEP  698 H= 698:00 +CPU= 19.121
  01:36:13 STEP  699 H= 699:00 +CPU= 31.385
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:37:46 STEP  700 H= 700:00 +CPU= 20.515
  01:38:05 STEP  701 H= 701:00 +CPU= 18.893
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:39:40 STEP  702 H= 702:00 +CPU= 32.859
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:41:20 STEP  703 H= 703:00 +CPU= 19.804
  01:41:40 STEP  704 H= 704:00 +CPU= 20.530
  01:42:11 STEP  705 H= 705:00 +CPU= 30.874
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:43:43 STEP  706 H= 706:00 +CPU= 20.424
  01:44:02 STEP  707 H= 707:00 +CPU= 19.436
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:45:49 STEP  708 H= 708:00 +CPU= 35.029
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000708
  01:46:08 STEP  709 H= 709:00 +CPU= 19.813
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:47:40 STEP  710 H= 710:00 +CPU= 20.264
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:49:13 STEP  711 H= 711:00 +CPU= 31.690
  01:49:33 STEP  712 H= 712:00 +CPU= 20.674
  01:49:53 STEP  713 H= 713:00 +CPU= 19.297
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:51:36 STEP  714 H= 714:00 +CPU= 32.248
  01:51:56 STEP  715 H= 715:00 +CPU= 19.590
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:53:46 STEP  716 H= 716:00 +CPU= 20.172
  01:54:17 STEP  717 H= 717:00 +CPU= 31.269
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:56:09 STEP  718 H= 718:00 +CPU= 20.236
  01:56:29 STEP  719 H= 719:00 +CPU= 20.118
  01:57:06 STEP  720 H= 720:00 +CPU= 37.549
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204_29.zip
Uploading the intermediate file: upload_file_29.zip
  01:57:25 STEP  721 H= 721:00 +CPU= 19.004
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:58:56 STEP  722 H= 722:00 +CPU= 19.914
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:00:29 STEP  723 H= 723:00 +CPU= 31.483
  02:00:49 STEP  724 H= 724:00 +CPU= 20.163
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:02:21 STEP  725 H= 725:00 +CPU= 20.373
  02:02:53 STEP  726 H= 726:00 +CPU= 31.960
  02:03:13 STEP  727 H= 727:00 +CPU= 20.235
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:04:55 STEP  728 H= 728:00 +CPU= 20.829
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:06:38 STEP  729 H= 729:00 +CPU= 31.378
  02:06:59 STEP  730 H= 730:00 +CPU= 21.319
  02:07:19 STEP  731 H= 731:00 +CPU= 19.859
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:08:54 STEP  732 H= 732:00 +CPU= 35.081
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000732
  02:09:14 STEP  733 H= 733:00 +CPU= 20.109
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:10:46 STEP  734 H= 734:00 +CPU= 20.372
  02:11:17 STEP  735 H= 735:00 +CPU= 31.835
  02:11:38 STEP  736 H= 736:00 +CPU= 20.783
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:13:00 STEP  737 H= 737:00 +CPU= 20.243
  02:13:32 STEP  738 H= 738:00 +CPU= 32.069
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:15:14 STEP  739 H= 739:00 +CPU= 20.114
  02:15:35 STEP  740 H= 740:00 +CPU= 20.736
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:16:59 STEP  741 H= 741:00 +CPU= 31.816
  02:17:19 STEP  742 H= 742:00 +CPU= 20.339
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:19:02 STEP  743 H= 743:00 +CPU= 20.629
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:20:29 STEP  744 H= 744:00 +CPU= 36.616
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204_30.zip
Uploading the intermediate file: upload_file_30.zip
  02:20:49 STEP  745 H= 745:00 +CPU= 20.061
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0070
(argv4) batchid: 962
(argv5) wuid: 12178714
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/16
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_56435e71a077fe9b66cece1d0898629b to: /var/lib/boinc-client/slots/16/oifs_43r3_ps_0070_1993050100_123_962_12178714.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/16/oifs_43r3_ps_0070_1993050100_123_962_12178714.zip
ic_ancil_file: ic_ancil_12178714
ifsdata_file: ifsdata_12178714
climate_data_file: clim_data_12178714
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5aa645fb931939c92f4159ccf1c5f718 to: /var/lib/boinc-client/slots/16/ic_ancil_12178714.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/16/ic_ancil_12178714.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4509268e533680c53f92685772cbe318 to: /var/lib/boinc-client/slots/16/ifsdata/ifsdata_12178714.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/16/ifsdata/ifsdata_12178714.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/16/159l_2/clim_data_12178714.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/16/159l_2/clim_data_12178714.zip
Checking for progress XML file: /var/lib/boinc-client/slots/16/progress_file_12178714.xml
Opened progress file ok : /var/lib/boinc-client/slots/16/progress_file_12178714.xml
-- Model is restarting --
Adjusting last_iter, 745, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/16/progress_file_12178714.xml
last_cpu_time: 20531
upload_file_number: 31
last_iter: 743
last_upload: 2678400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204
The child process has been launched with process id: 368109
Executing the command: /var/lib/boinc-client/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:368109:368109] [20230121:022405:1674235445:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:24:44 STEP  744 H= 744:00 +CPU= 36.122
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:26:25 STEP  745 H= 745:00 +CPU= 19.954
  02:26:46 STEP  746 H= 746:00 +CPU= 21.031
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:28:09 STEP  747 H= 747:00 +CPU= 32.054
  02:28:30 STEP  748 H= 748:00 +CPU= 20.894
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:30:01 STEP  749 H= 749:00 +CPU= 19.841
  02:30:33 STEP  750 H= 750:00 +CPU= 32.344
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:31:55 STEP  751 H= 751:00 +CPU= 20.456
  02:32:16 STEP  752 H= 752:00 +CPU= 21.288
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:33:48 STEP  753 H= 753:00 +CPU= 31.749
  02:34:08 STEP  754 H= 754:00 +CPU= 20.037
  02:34:28 STEP  755 H= 755:00 +CPU= 19.569
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:36:24 STEP  756 H= 756:00 +CPU= 35.044
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000756
  02:36:44 STEP  757 H= 757:00 +CPU= 19.739
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:38:06 STEP  758 H= 758:00 +CPU= 20.627
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:39:30 STEP  759 H= 759:00 +CPU= 31.865
  02:39:51 STEP  760 H= 760:00 +CPU= 20.434
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:41:12 STEP  761 H= 761:00 +CPU= 19.868
  02:41:44 STEP  762 H= 762:00 +CPU= 32.173
  02:42:04 STEP  763 H= 763:00 +CPU= 19.958
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:43:46 STEP  764 H= 764:00 +CPU= 20.328
  02:44:17 STEP  765 H= 765:00 +CPU= 31.498
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:45:59 STEP  766 H= 766:00 +CPU= 20.229
  02:46:19 STEP  767 H= 767:00 +CPU= 20.888
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:47:48 STEP  768 H= 768:00 +CPU= 37.227
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/16/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178714/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0070_1993050100_123_962_12178714_2_r1883353204_31.zip
Uploading the intermediate file: upload_file_31.zip
  02:48:08 STEP  769 H= 769:00 +CPU= 19.763
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:49:40 STEP  770 H= 770:00 +CPU= 20.425
  02:50:12 STEP  771 H= 771:00 +CPU= 31.576
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:51:45 STEP  772 H= 772:00 +CPU= 21.184
  02:52:06 STEP  773 H= 773:00 +CPU= 20.429
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:53:38 STEP  774 H= 774:00 +CPU= 32.431
  02:53:58 STEP  775 H= 775:00 +CPU= 19.862
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net