climateprediction.net home page
Task 22268889

Task 22268889

Name oifs_43r3_ps_0050_1998050100_123_967_12183694_0
Workunit 12183694
Created 22 Dec 2022, 15:11:19 UTC
Sent 10 Jan 2023, 12:53:10 UTC
Report deadline 9 Feb 2023, 12:53:10 UTC
Received 19 Jan 2023, 11:30:55 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1534826
Run time 13 hours 51 min 26 sec
CPU time 12 hours 28 min 31 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,769.32 MB
Peak swap size 4,978.08 MB
Peak disk usage 5,279.53 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
t/oifs_43r3_ps_12183694/ICMGGhq0f+002736
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002736
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002736
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002748
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002748
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002748
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0050_1998050100_123_967_12183694_0_r1101536207_114.zip
Uploading the intermediate file: upload_file_114.zip
  16:07:31 STEP 2761 H=2761:00 +CPU= 12.854
  16:07:44 STEP 2762 H=2762:00 +CPU= 13.172
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:08:10 STEP 2763 H=2763:00 +CPU= 21.082
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:08:28 STEP 2764 H=2764:00 +CPU= 13.862
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:08:43 STEP 2765 H=2765:00 +CPU= 13.091
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:09:11 STEP 2766 H=2766:00 +CPU= 22.261
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:09:29 STEP 2767 H=2767:00 +CPU= 13.401
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:09:48 STEP 2768 H=2768:00 +CPU= 14.410
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:10:15 STEP 2769 H=2769:00 +CPU= 22.223
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:10:34 STEP 2770 H=2770:00 +CPU= 14.183
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:10:51 STEP 2771 H=2771:00 +CPU= 13.283
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:11:21 STEP 2772 H=2772:00 +CPU= 24.075
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/2/ICMGGhq0f+002772
Moving to projects directory: /root/slots/2/ICMSHhq0f+002772
Moving to projects directory: /root/slots/2/ICMUAhq0f+002772
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:11:39 STEP 2773 H=2773:00 +CPU= 13.426
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:11:57 STEP 2774 H=2774:00 +CPU= 13.938
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:12:23 STEP 2775 H=2775:00 +CPU= 21.927
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:12:41 STEP 2776 H=2776:00 +CPU= 13.695
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:12:59 STEP 2777 H=2777:00 +CPU= 13.436
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:13:27 STEP 2778 H=2778:00 +CPU= 22.314
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:13:43 STEP 2779 H=2779:00 +CPU= 13.680
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:14:02 STEP 2780 H=2780:00 +CPU= 14.066
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:14:30 STEP 2781 H=2781:00 +CPU= 21.785
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:14:48 STEP 2782 H=2782:00 +CPU= 13.930
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:15:04 STEP 2783 H=2783:00 +CPU= 13.597
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:15:38 STEP 2784 H=2784:00 +CPU= 25.539
Moving to projects directory: /root/slots/2/ICMGGhq0f+002784
Moving to projects directory: /root/slots/2/ICMSHhq0f+002784
Moving to projects directory: /root/slots/2/ICMUAhq0f+002784
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002760
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002760
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002760
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002772
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002772
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002772
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0050_1998050100_123_967_12183694_0_r1101536207_115.zip
Uploading the intermediate file: upload_file_115.zip
  16:15:52 STEP 2785 H=2785:00 +CPU= 13.796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:16:08 STEP 2786 H=2786:00 +CPU= 13.776
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:16:37 STEP 2787 H=2787:00 +CPU= 22.341
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:16:54 STEP 2788 H=2788:00 +CPU= 14.222
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:17:11 STEP 2789 H=2789:00 +CPU= 13.441
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:17:40 STEP 2790 H=2790:00 +CPU= 22.462
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:17:58 STEP 2791 H=2791:00 +CPU= 13.527
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:18:14 STEP 2792 H=2792:00 +CPU= 14.127
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:18:43 STEP 2793 H=2793:00 +CPU= 21.929
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:19:01 STEP 2794 H=2794:00 +CPU= 14.110
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:19:19 STEP 2795 H=2795:00 +CPU= 13.319
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:19:49 STEP 2796 H=2796:00 +CPU= 24.234
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/2/ICMGGhq0f+002796
Moving to projects directory: /root/slots/2/ICMSHhq0f+002796
Moving to projects directory: /root/slots/2/ICMUAhq0f+002796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:20:07 STEP 2797 H=2797:00 +CPU= 13.689
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:20:23 STEP 2798 H=2798:00 +CPU= 13.880
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:20:51 STEP 2799 H=2799:00 +CPU= 21.784
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:21:09 STEP 2800 H=2800:00 +CPU= 13.533
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:21:25 STEP 2801 H=2801:00 +CPU= 13.304
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:21:53 STEP 2802 H=2802:00 +CPU= 21.848
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:22:11 STEP 2803 H=2803:00 +CPU= 13.594
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:22:29 STEP 2804 H=2804:00 +CPU= 13.743
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:22:58 STEP 2805 H=2805:00 +CPU= 22.255
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:23:15 STEP 2806 H=2806:00 +CPU= 14.204
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:23:33 STEP 2807 H=2807:00 +CPU= 13.860
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:24:08 STEP 2808 H=2808:00 +CPU= 25.755
Moving to projects directory: /root/slots/2/ICMGGhq0f+002808
Moving to projects directory: /root/slots/2/ICMSHhq0f+002808
Moving to projects directory: /root/slots/2/ICMUAhq0f+002808
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002784
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002784
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002784
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002796
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002796
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002796
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0050_1998050100_123_967_12183694_0_r1101536207_116.zip
Uploading the intermediate file: upload_file_116.zip
  16:24:21 STEP 2809 H=2809:00 +CPU= 13.196
  16:24:35 STEP 2810 H=2810:00 +CPU= 13.553
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:25:01 STEP 2811 H=2811:00 +CPU= 21.017
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:18 STEP 2812 H=2812:00 +CPU= 35.710
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:41 STEP 2813 H=2813:00 +CPU= 11.642
Suspend request received from the BOINC client, suspending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1998050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0050
(argv4) batchid: 967
(argv5) wuid: 12183694
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/2
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12183694
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_020c67d2f0fd2707ee82fdf7cbdaa809 to: /root/slots/2/oifs_43r3_ps_0050_1998050100_123_967_12183694.zip
Unzipping the namelist zip file: /root/slots/2/oifs_43r3_ps_0050_1998050100_123_967_12183694.zip
ic_ancil_file: ic_ancil_12183694
ifsdata_file: ifsdata_12183694
climate_data_file: clim_data_12183694
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2f633ee99ec703df0a7d7346735f3e4c to: /root/slots/2/ic_ancil_12183694.zip
Unzipping the IC ancils zip file: /root/slots/2/ic_ancil_12183694.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_558e53786ff4f010f5eb66d41b134bc3 to: /root/slots/2/ifsdata/ifsdata_12183694.zip
Unzipping the ifsdata_zip file: /root/slots/2/ifsdata/ifsdata_12183694.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/2/159l_2/clim_data_12183694.zip
Unzipping the climate data zip file: /root/slots/2/159l_2/clim_data_12183694.zip
Checking for progress XML file: /root/slots/2/progress_file_12183694.xml
Opened progress file ok : /root/slots/2/progress_file_12183694.xml
-- Model is restarting --
Adjusting last_iter, 2813, to previous model restart step.
Creating progress file: /root/slots/2/progress_file_12183694.xml
last_cpu_time: 42805
upload_file_number: 117
last_iter: 2807
last_upload: 10108800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0050_1998050100_123_967_12183694_0_r1101536207
The child process has been launched with process id: 3305
Executing the command: /root/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:50-privat:1:1:3305:3305] [20230119:113012:1674124212:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:30:39 STEP 2808 H=2808:00 +CPU= 20.839
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:30:55 STEP 2809 H=2809:00 +CPU= 11.828
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:31:08 STEP 2810 H=2810:00 +CPU= 11.433
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:31:30 STEP 2811 H=2811:00 +CPU= 17.685
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:31:47 STEP 2812 H=2812:00 +CPU= 12.685
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:32:03 STEP 2813 H=2813:00 +CPU= 11.660
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:32:26 STEP 2814 H=2814:00 +CPU= 18.744
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:32:39 STEP 2815 H=2815:00 +CPU= 11.053
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:32:54 STEP 2816 H=2816:00 +CPU= 11.566
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:33:17 STEP 2817 H=2817:00 +CPU= 18.358
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:33:33 STEP 2818 H=2818:00 +CPU= 11.609
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:33:46 STEP 2819 H=2819:00 +CPU= 11.109
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:34:10 STEP 2820 H=2820:00 +CPU= 20.495
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/2/ICMGGhq0f+002820
Moving to projects directory: /root/slots/2/ICMSHhq0f+002820
Moving to projects directory: /root/slots/2/ICMUAhq0f+002820
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:34:26 STEP 2821 H=2821:00 +CPU= 11.594
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:34:40 STEP 2822 H=2822:00 +CPU= 11.744
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:04 STEP 2823 H=2823:00 +CPU= 18.379
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:18 STEP 2824 H=2824:00 +CPU= 11.490
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:33 STEP 2825 H=2825:00 +CPU= 11.169
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:56 STEP 2826 H=2826:00 +CPU= 18.708
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:36:10 STEP 2827 H=2827:00 +CPU= 12.194
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:36:27 STEP 2828 H=2828:00 +CPU= 12.342
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:36:53 STEP 2829 H=2829:00 +CPU= 19.764
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:37:07 STEP 2830 H=2830:00 +CPU= 12.120
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:37:24 STEP 2831 H=2831:00 +CPU= 12.329
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:37:50 STEP 2832 H=2832:00 +CPU= 21.893
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/2/ICMGGhq0f+002832
Moving to projects directory: /root/slots/2/ICMSHhq0f+002832
Moving to projects directory: /root/slots/2/ICMUAhq0f+002832
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002808
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002808
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002808
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002820
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002820
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002820
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0050_1998050100_123_967_12183694_0_r1101536207_117.zip
Uploading the intermediate file: upload_file_117.zip
  11:38:03 STEP 2833 H=2833:00 +CPU= 11.039
  11:38:14 STEP 2834 H=2834:00 +CPU= 11.855
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:38:37 STEP 2835 H=2835:00 +CPU= 18.825
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:38:55 STEP 2836 H=2836:00 +CPU= 11.702
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:39:11 STEP 2837 H=2837:00 +CPU= 12.007
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:39:35 STEP 2838 H=2838:00 +CPU= 19.696
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:39:51 STEP 2839 H=2839:00 +CPU= 11.547
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:40:05 STEP 2840 H=2840:00 +CPU= 12.225
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:40:30 STEP 2841 H=2841:00 +CPU= 19.522
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:40:45 STEP 2842 H=2842:00 +CPU= 12.189
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:41:00 STEP 2843 H=2843:00 +CPU= 11.394
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:41:28 STEP 2844 H=2844:00 +CPU= 21.610
Moving to projects directory: /root/slots/2/ICMGGhq0f+002844
Moving to projects directory: /root/slots/2/ICMSHhq0f+002844
Moving to projects directory: /root/slots/2/ICMUAhq0f+002844
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:41:41 STEP 2845 H=2845:00 +CPU= 11.568
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:41:58 STEP 2846 H=2846:00 +CPU= 12.129
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:42:21 STEP 2847 H=2847:00 +CPU= 18.935
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:42:35 STEP 2848 H=2848:00 +CPU= 12.104
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:42:50 STEP 2849 H=2849:00 +CPU= 11.531
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:43:12 STEP 2850 H=2850:00 +CPU= 19.208
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:43:27 STEP 2851 H=2851:00 +CPU= 11.824
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:43:42 STEP 2852 H=2852:00 +CPU= 12.102
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:44:03 STEP 2853 H=2853:00 +CPU= 19.183
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:44:19 STEP 2854 H=2854:00 +CPU= 11.862
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:44:33 STEP 2855 H=2855:00 +CPU= 12.600
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:44:59 STEP 2856 H=2856:00 +CPU= 22.092
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/2/ICMGGhq0f+002856
Moving to projects directory: /root/slots/2/ICMSHhq0f+002856
Moving to projects directory: /root/slots/2/ICMUAhq0f+002856
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002832
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002832
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002832
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002844
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002844
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002844
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0050_1998050100_123_967_12183694_0_r1101536207_118.zip
Uploading the intermediate file: upload_file_118.zip
  11:45:13 STEP 2857 H=2857:00 +CPU= 11.744
  11:45:25 STEP 2858 H=2858:00 +CPU= 11.978
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:45:48 STEP 2859 H=2859:00 +CPU= 18.519
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:46:02 STEP 2860 H=2860:00 +CPU= 12.154
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:46:18 STEP 2861 H=2861:00 +CPU= 11.915
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:46:42 STEP 2862 H=2862:00 +CPU= 19.744
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:46:57 STEP 2863 H=2863:00 +CPU= 11.482
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:47:11 STEP 2864 H=2864:00 +CPU= 12.045
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:47:35 STEP 2865 H=2865:00 +CPU= 19.375
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:47:51 STEP 2866 H=2866:00 +CPU= 11.837
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:48:05 STEP 2867 H=2867:00 +CPU= 12.084
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:48:32 STEP 2868 H=2868:00 +CPU= 20.945
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/2/ICMGGhq0f+002868
Moving to projects directory: /root/slots/2/ICMSHhq0f+002868
Moving to projects directory: /root/slots/2/ICMUAhq0f+002868
  11:48:45 STEP 2869 H=2869:00 +CPU= 11.528
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:49:01 STEP 2870 H=2870:00 +CPU= 12.191
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:49:25 STEP 2871 H=2871:00 +CPU= 19.096
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:49:40 STEP 2872 H=2872:00 +CPU= 11.780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:49:54 STEP 2873 H=2873:00 +CPU= 11.688
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:50:19 STEP 2874 H=2874:00 +CPU= 18.965
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:50:33 STEP 2875 H=2875:00 +CPU= 11.532
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:50:49 STEP 2876 H=2876:00 +CPU= 12.040
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:51:12 STEP 2877 H=2877:00 +CPU= 18.915
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:51:28 STEP 2878 H=2878:00 +CPU= 12.073
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:51:42 STEP 2879 H=2879:00 +CPU= 12.203
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:52:10 STEP 2880 H=2880:00 +CPU= 22.338
Moving to projects directory: /root/slots/2/ICMGGhq0f+002880
Moving to projects directory: /root/slots/2/ICMSHhq0f+002880
Moving to projects directory: /root/slots/2/ICMUAhq0f+002880
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002856
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002856
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002856
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002868
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002868
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002868
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0050_1998050100_123_967_12183694_0_r1101536207_119.zip
Uploading the intermediate file: upload_file_119.zip
  11:52:22 STEP 2881 H=2881:00 +CPU= 11.598
  11:52:34 STEP 2882 H=2882:00 +CPU= 11.780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:52:59 STEP 2883 H=2883:00 +CPU= 18.997
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:53:13 STEP 2884 H=2884:00 +CPU= 12.154
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:53:29 STEP 2885 H=2885:00 +CPU= 11.674
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:53:52 STEP 2886 H=2886:00 +CPU= 19.725
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:54:08 STEP 2887 H=2887:00 +CPU= 11.385
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:54:22 STEP 2888 H=2888:00 +CPU= 11.916
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:54:45 STEP 2889 H=2889:00 +CPU= 18.803
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:55:01 STEP 2890 H=2890:00 +CPU= 11.931
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:55:17 STEP 2891 H=2891:00 +CPU= 12.038
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:55:42 STEP 2892 H=2892:00 +CPU= 21.204
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/2/ICMGGhq0f+002892
Moving to projects directory: /root/slots/2/ICMSHhq0f+002892
Moving to projects directory: /root/slots/2/ICMUAhq0f+002892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:55:58 STEP 2893 H=2893:00 +CPU= 11.715
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:56:12 STEP 2894 H=2894:00 +CPU= 12.311
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:56:37 STEP 2895 H=2895:00 +CPU= 19.091
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:56:51 STEP 2896 H=2896:00 +CPU= 11.974
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:57:07 STEP 2897 H=2897:00 +CPU= 11.862
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:57:31 STEP 2898 H=2898:00 +CPU= 19.616
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:57:44 STEP 2899 H=2899:00 +CPU= 11.613
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:58:01 STEP 2900 H=2900:00 +CPU= 12.367
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:58:24 STEP 2901 H=2901:00 +CPU= 19.412
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:58:40 STEP 2902 H=2902:00 +CPU= 12.095
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:58:54 STEP 2903 H=2903:00 +CPU= 11.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:59:23 STEP 2904 H=2904:00 +CPU= 22.573
Moving to projects directory: /root/slots/2/ICMGGhq0f+002904
Moving to projects directory: /root/slots/2/ICMSHhq0f+002904
Moving to projects directory: /root/slots/2/ICMUAhq0f+002904
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002880
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002880
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002880
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002892
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002892
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002892
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0050_1998050100_123_967_12183694_0_r1101536207_120.zip
Uploading the intermediate file: upload_file_120.zip
  11:59:35 STEP 2905 H=2905:00 +CPU= 11.800
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:59:48 STEP 2906 H=2906:00 +CPU= 11.664
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:00:12 STEP 2907 H=2907:00 +CPU= 19.781
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:00:28 STEP 2908 H=2908:00 +CPU= 12.218
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:00:42 STEP 2909 H=2909:00 +CPU= 11.862
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:01:08 STEP 2910 H=2910:00 +CPU= 19.638
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:01:21 STEP 2911 H=2911:00 +CPU= 11.358
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:01:38 STEP 2912 H=2912:00 +CPU= 12.116
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:02:00 STEP 2913 H=2913:00 +CPU= 18.666
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:02:14 STEP 2914 H=2914:00 +CPU= 11.953
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:02:30 STEP 2915 H=2915:00 +CPU= 11.868
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:02:58 STEP 2916 H=2916:00 +CPU= 21.299
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/2/ICMGGhq0f+002916
Moving to projects directory: /root/slots/2/ICMSHhq0f+002916
Moving to projects directory: /root/slots/2/ICMUAhq0f+002916
  12:03:11 STEP 2917 H=2917:00 +CPU= 11.701
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:03:27 STEP 2918 H=2918:00 +CPU= 11.901
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:03:50 STEP 2919 H=2919:00 +CPU= 18.910
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:04:04 STEP 2920 H=2920:00 +CPU= 11.959
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:04:20 STEP 2921 H=2921:00 +CPU= 11.606
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:04:43 STEP 2922 H=2922:00 +CPU= 19.344
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:04:59 STEP 2923 H=2923:00 +CPU= 11.655
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:05:13 STEP 2924 H=2924:00 +CPU= 12.075
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:05:38 STEP 2925 H=2925:00 +CPU= 18.731
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:05:52 STEP 2926 H=2926:00 +CPU= 11.874
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:06:08 STEP 2927 H=2927:00 +CPU= 12.093
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:06:34 STEP 2928 H=2928:00 +CPU= 22.505
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/2/ICMGGhq0f+002928
Moving to projects directory: /root/slots/2/ICMSHhq0f+002928
Moving to projects directory: /root/slots/2/ICMUAhq0f+002928
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002904
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002904
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002904
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002916
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002916
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002916
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0050_1998050100_123_967_12183694_0_r1101536207_121.zip
Uploading the intermediate file: upload_file_121.zip
  12:06:48 STEP 2929 H=2929:00 +CPU= 11.490
  12:07:00 STEP 2930 H=2930:00 +CPU= 11.741
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:07:23 STEP 2931 H=2931:00 +CPU= 19.232
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:07:39 STEP 2932 H=2932:00 +CPU= 11.952
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:07:53 STEP 2933 H=2933:00 +CPU= 11.631
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:08:18 STEP 2934 H=2934:00 +CPU= 19.479
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:08:31 STEP 2935 H=2935:00 +CPU= 11.405
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:08:47 STEP 2936 H=2936:00 +CPU= 11.989
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:09:10 STEP 2937 H=2937:00 +CPU= 18.905
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:09:25 STEP 2938 H=2938:00 +CPU= 12.044
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:09:40 STEP 2939 H=2939:00 +CPU= 11.794
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:10:08 STEP 2940 H=2940:00 +CPU= 21.423
Moving to projects directory: /root/slots/2/ICMGGhq0f+002940
Moving to projects directory: /root/slots/2/ICMSHhq0f+002940
Moving to projects directory: /root/slots/2/ICMUAhq0f+002940
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:10:22 STEP 2941 H=2941:00 +CPU= 11.689
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:10:38 STEP 2942 H=2942:00 +CPU= 12.105
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:11:01 STEP 2943 H=2943:00 +CPU= 19.271
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:11:17 STEP 2944 H=2944:00 +CPU= 11.784
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:11:30 STEP 2945 H=2945:00 +CPU= 11.453
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:11:54 STEP 2946 H=2946:00 +CPU= 19.493
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:12:10 STEP 2947 H=2947:00 +CPU= 11.623
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:12:24 STEP 2948 H=2948:00 +CPU= 12.108
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:12:49 STEP 2949 H=2949:00 +CPU= 18.852
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:13:02 STEP 2950 H=2950:00 +CPU= 11.844
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:13:18 STEP 2951 H=2951:00 +CPU= 12.017
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:13:44 STEP 2952 H=2952:00 +CPU= 21.920
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
The child process terminated with status: 0
Moving to projects directory: /root/slots/2/ICMGGhq0f+002952
Moving to projects directory: /root/slots/2/ICMSHhq0f+002952
Moving to projects directory: /root/slots/2/ICMUAhq0f+002952
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002928
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002928
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002928
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002940
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002940
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002940
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMGGhq0f+002952
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMSHhq0f+002952
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12183694/ICMUAhq0f+002952
Zipping up the final file: /root/projects/climateprediction.net/oifs_43r3_ps_0050_1998050100_123_967_12183694_0_r1101536207_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
12:17:09 (3268): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:28:26 1534826 22268889 oifs_43r3_ps_0050_1998050100_123_967_12183694_0 10,623,600 44,911 0.0042
09 May 2023 16:24:22 1534826 22268889 oifs_43r3_ps_0050_1998050100_123_967_12183694_0 9,504,000 39,942 0.0042
09 May 2023 16:24:12 1534826 22268889 oifs_43r3_ps_0050_1998050100_123_967_12183694_0 8,640,000 36,167 0.0042
09 May 2023 16:22:29 1534826 22268889 oifs_43r3_ps_0050_1998050100_123_967_12183694_0 6,998,400 28,877 0.0041
09 May 2023 16:22:23 1534826 22268889 oifs_43r3_ps_0050_1998050100_123_967_12183694_0 5,961,600 24,412 0.0041
09 May 2023 16:16:52 1534826 22268889 oifs_43r3_ps_0050_1998050100_123_967_12183694_0 5,097,600 20,815 0.0041
09 May 2023 16:16:46 1534826 22268889 oifs_43r3_ps_0050_1998050100_123_967_12183694_0 4,233,600 17,146 0.0040
09 May 2023 16:15:39 1534826 22268889 oifs_43r3_ps_0050_1998050100_123_967_12183694_0 2,937,600 11,489 0.0039
09 May 2023 16:14:48 1534826 22268889 oifs_43r3_ps_0050_1998050100_123_967_12183694_0 1,728,000 6,228 0.0036
09 May 2023 16:14:46 1534826 22268889 oifs_43r3_ps_0050_1998050100_123_967_12183694_0 864,000 3,262 0.0038


©2024 climateprediction.net