climateprediction.net home page
Task 22259336

Task 22259336

Name oifs_43r3_ps_0023_1989050100_123_958_12174667_0
Workunit 12174667
Created 21 Dec 2022, 15:27:31 UTC
Sent 21 Dec 2022, 17:10:47 UTC
Report deadline 20 Jan 2023, 17:10:47 UTC
Received 24 Dec 2022, 1:28:53 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1531398
Run time 23 hours 46 min 57 sec
CPU time 18 hours 34 min 27 sec
Validate state Invalid
Credit 13,789.44
Device peak FLOPS 7.99 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,787.32 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,178.29 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
17 STEP 2485 H=2485:00 +CPU= 23.528
  22:15:43 STEP 2486 H=2486:00 +CPU= 23.725
  22:16:19 STEP 2487 H=2487:00 +CPU= 34.984
  22:16:44 STEP 2488 H=2488:00 +CPU= 23.693
  22:17:07 STEP 2489 H=2489:00 +CPU= 22.859
  22:17:43 STEP 2490 H=2490:00 +CPU= 34.455
  22:18:07 STEP 2491 H=2491:00 +CPU= 23.097
  22:18:32 STEP 2492 H=2492:00 +CPU= 24.086
  22:19:09 STEP 2493 H=2493:00 +CPU= 34.728
  22:19:34 STEP 2494 H=2494:00 +CPU= 24.330
  22:20:00 STEP 2495 H=2495:00 +CPU= 24.895
  22:20:46 STEP 2496 H=2496:00 +CPU= 41.350
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_82.zip
Uploading the intermediate file: upload_file_82.zip
  22:21:12 STEP 2497 H=2497:00 +CPU= 22.094
  22:21:39 STEP 2498 H=2498:00 +CPU= 23.413
  22:22:17 STEP 2499 H=2499:00 +CPU= 32.246
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0023
(argv4) batchid: 958
(argv5) wuid: 12174667
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/21
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7db76cc0ef87c52e0949750383e2c0fb to: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip
ic_ancil_file: ic_ancil_12174667
ifsdata_file: ifsdata_12174667
climate_data_file: clim_data_12174667
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7d56b2d96fd43574459082e0fe824891 to: /var/lib/boinc/slots/21/ic_ancil_12174667.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/21/ic_ancil_12174667.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3f05a2cabd27cda893615872183ddf12 to: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip
Checking for progress XML file: /var/lib/boinc/slots/21/progress_file_12174667.xml
Opened progress file ok : /var/lib/boinc/slots/21/progress_file_12174667.xml
-- Model is restarting --
Adjusting last_iter, 2500, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/21/progress_file_12174667.xml
last_cpu_time: 57265
upload_file_number: 83
last_iter: 2495
last_upload: 8985600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911
The child process has been launched with process id: 6332
Executing the command: /var/lib/boinc/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0023
(argv4) batchid: 958
(argv5) wuid: 12174667
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/21
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7db76cc0ef87c52e0949750383e2c0fb to: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip
ic_ancil_file: ic_ancil_12174667
ifsdata_file: ifsdata_12174667
climate_data_file: clim_data_12174667
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7d56b2d96fd43574459082e0fe824891 to: /var/lib/boinc/slots/21/ic_ancil_12174667.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/21/ic_ancil_12174667.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3f05a2cabd27cda893615872183ddf12 to: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip
Checking for progress XML file: /var/lib/boinc/slots/21/progress_file_12174667.xml
Opened progress file ok : /var/lib/boinc/slots/21/progress_file_12174667.xml
-- Model is restarting --
Adjusting last_iter, 2496, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/21/progress_file_12174667.xml
last_cpu_time: 57273
upload_file_number: 83
last_iter: 2495
last_upload: 8985600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911
The child process has been launched with process id: 7022
Executing the command: /var/lib/boinc/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:27:20 STEP 2496 H=2496:00 +CPU= 41.051
  12:27:43 STEP 2497 H=2497:00 +CPU= 23.202
  12:28:02 STEP 2498 H=2498:00 +CPU= 18.403
  12:28:30 STEP 2499 H=2499:00 +CPU= 27.736
  12:28:52 STEP 2500 H=2500:00 +CPU= 20.312
  12:29:14 STEP 2501 H=2501:00 +CPU= 20.336
  12:29:47 STEP 2502 H=2502:00 +CPU= 31.494
  12:30:06 STEP 2503 H=2503:00 +CPU= 19.552
  12:30:28 STEP 2504 H=2504:00 +CPU= 21.196
  12:31:00 STEP 2505 H=2505:00 +CPU= 30.987
  12:31:23 STEP 2506 H=2506:00 +CPU= 22.508
  12:31:46 STEP 2507 H=2507:00 +CPU= 21.610
  12:32:33 STEP 2508 H=2508:00 +CPU= 39.900
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002508
  12:33:00 STEP 2509 H=2509:00 +CPU= 23.901
  12:33:32 STEP 2510 H=2510:00 +CPU= 25.415
  12:34:13 STEP 2511 H=2511:00 +CPU= 35.850
  12:34:44 STEP 2512 H=2512:00 +CPU= 25.809
  12:35:11 STEP 2513 H=2513:00 +CPU= 23.852
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:36:09 STEP 2514 H=2514:00 +CPU= 35.505
  12:36:42 STEP 2515 H=2515:00 +CPU= 24.262
  12:37:15 STEP 2516 H=2516:00 +CPU= 25.048
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:38:35 STEP 2517 H=2517:00 +CPU= 34.355
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:39:31 STEP 2518 H=2518:00 +CPU= 24.560
  12:40:04 STEP 2519 H=2519:00 +CPU= 25.991
  12:40:51 STEP 2520 H=2520:00 +CPU= 40.589
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_83.zip
Uploading the intermediate file: upload_file_83.zip
  12:41:17 STEP 2521 H=2521:00 +CPU= 22.433
  12:41:47 STEP 2522 H=2522:00 +CPU= 23.619
  12:42:26 STEP 2523 H=2523:00 +CPU= 33.358
  12:42:56 STEP 2524 H=2524:00 +CPU= 23.651
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:43:35 STEP 2525 H=2525:00 +CPU= 22.086
  12:44:18 STEP 2526 H=2526:00 +CPU= 34.846
  12:44:47 STEP 2527 H=2527:00 +CPU= 23.839
  12:45:16 STEP 2528 H=2528:00 +CPU= 23.839
  12:45:57 STEP 2529 H=2529:00 +CPU= 34.135
  12:46:27 STEP 2530 H=2530:00 +CPU= 24.178
  12:46:53 STEP 2531 H=2531:00 +CPU= 22.459
  12:47:30 STEP 2532 H=2532:00 +CPU= 32.480
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002532
  12:47:54 STEP 2533 H=2533:00 +CPU= 19.704
  12:48:22 STEP 2534 H=2534:00 +CPU= 21.399
  12:48:57 STEP 2535 H=2535:00 +CPU= 29.170
  12:49:21 STEP 2536 H=2536:00 +CPU= 19.893
  12:49:44 STEP 2537 H=2537:00 +CPU= 19.335
  12:50:24 STEP 2538 H=2538:00 +CPU= 30.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:51:17 STEP 2539 H=2539:00 +CPU= 22.693
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:51:56 STEP 2540 H=2540:00 +CPU= 23.463
  12:52:38 STEP 2541 H=2541:00 +CPU= 34.556
  12:53:07 STEP 2542 H=2542:00 +CPU= 24.133
  12:53:38 STEP 2543 H=2543:00 +CPU= 24.807
  12:54:28 STEP 2544 H=2544:00 +CPU= 40.471
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_84.zip
Uploading the intermediate file: upload_file_84.zip
  12:54:58 STEP 2545 H=2545:00 +CPU= 23.902
  12:55:27 STEP 2546 H=2546:00 +CPU= 23.799
  12:56:09 STEP 2547 H=2547:00 +CPU= 34.164
  12:56:38 STEP 2548 H=2548:00 +CPU= 23.936
  12:57:07 STEP 2549 H=2549:00 +CPU= 23.478
  12:57:49 STEP 2550 H=2550:00 +CPU= 35.137
  12:58:18 STEP 2551 H=2551:00 +CPU= 23.509
  12:58:46 STEP 2552 H=2552:00 +CPU= 23.644
  12:59:24 STEP 2553 H=2553:00 +CPU= 32.999
  12:59:50 STEP 2554 H=2554:00 +CPU= 21.693
  13:00:15 STEP 2555 H=2555:00 +CPU= 21.035
  13:00:59 STEP 2556 H=2556:00 +CPU= 35.991
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002556
  13:01:26 STEP 2557 H=2557:00 +CPU= 21.774
  13:01:48 STEP 2558 H=2558:00 +CPU= 20.370
  13:02:24 STEP 2559 H=2559:00 +CPU= 31.326
  13:02:51 STEP 2560 H=2560:00 +CPU= 22.068
  13:03:12 STEP 2561 H=2561:00 +CPU= 19.562
  13:03:51 STEP 2562 H=2562:00 +CPU= 32.501
  13:04:18 STEP 2563 H=2563:00 +CPU= 22.484
  13:04:46 STEP 2564 H=2564:00 +CPU= 22.678
  13:05:25 STEP 2565 H=2565:00 +CPU= 32.869
  13:05:53 STEP 2566 H=2566:00 +CPU= 22.926
  13:06:21 STEP 2567 H=2567:00 +CPU= 22.872
  13:07:08 STEP 2568 H=2568:00 +CPU= 39.038
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_85.zip
Uploading the intermediate file: upload_file_85.zip
  13:07:35 STEP 2569 H=2569:00 +CPU= 22.266
  13:08:02 STEP 2570 H=2570:00 +CPU= 22.303
  13:08:42 STEP 2571 H=2571:00 +CPU= 33.092
  13:09:09 STEP 2572 H=2572:00 +CPU= 22.518
  13:09:35 STEP 2573 H=2573:00 +CPU= 21.892
  13:10:12 STEP 2574 H=2574:00 +CPU= 32.170
  13:10:37 STEP 2575 H=2575:00 +CPU= 21.423
  13:11:02 STEP 2576 H=2576:00 +CPU= 21.224
  13:11:38 STEP 2577 H=2577:00 +CPU= 31.262
  13:12:05 STEP 2578 H=2578:00 +CPU= 22.561
  13:12:32 STEP 2579 H=2579:00 +CPU= 22.036
  13:13:15 STEP 2580 H=2580:00 +CPU= 36.650
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002580
  13:13:40 STEP 2581 H=2581:00 +CPU= 22.006
  13:14:05 STEP 2582 H=2582:00 +CPU= 22.210
  13:14:44 STEP 2583 H=2583:00 +CPU= 33.124
  13:15:11 STEP 2584 H=2584:00 +CPU= 23.393
  13:15:38 STEP 2585 H=2585:00 +CPU= 23.350
  13:16:19 STEP 2586 H=2586:00 +CPU= 35.109
  13:16:46 STEP 2587 H=2587:00 +CPU= 23.302
  13:17:13 STEP 2588 H=2588:00 +CPU= 23.187
  13:17:52 STEP 2589 H=2589:00 +CPU= 34.016
  13:18:21 STEP 2590 H=2590:00 +CPU= 23.734
  13:18:50 STEP 2591 H=2591:00 +CPU= 24.450
  13:19:37 STEP 2592 H=2592:00 +CPU= 40.114
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_86.zip
Uploading the intermediate file: upload_file_86.zip
  13:20:05 STEP 2593 H=2593:00 +CPU= 23.981
  13:20:35 STEP 2594 H=2594:00 +CPU= 24.658
  13:21:19 STEP 2595 H=2595:00 +CPU= 36.595
  13:21:48 STEP 2596 H=2596:00 +CPU= 24.123
  13:22:15 STEP 2597 H=2597:00 +CPU= 22.666
  13:22:57 STEP 2598 H=2598:00 +CPU= 36.503
  13:23:27 STEP 2599 H=2599:00 +CPU= 24.605
  13:23:57 STEP 2600 H=2600:00 +CPU= 25.490
  13:24:38 STEP 2601 H=2601:00 +CPU= 35.688
  13:25:09 STEP 2602 H=2602:00 +CPU= 25.711
  13:25:38 STEP 2603 H=2603:00 +CPU= 25.052
  13:26:27 STEP 2604 H=2604:00 +CPU= 42.470
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002604
  13:26:59 STEP 2605 H=2605:00 +CPU= 26.140
  13:27:33 STEP 2606 H=2606:00 +CPU= 27.057
  13:28:23 STEP 2607 H=2607:00 +CPU= 40.540
  13:28:57 STEP 2608 H=2608:00 +CPU= 27.784
  13:29:32 STEP 2609 H=2609:00 +CPU= 27.812
  13:30:24 STEP 2610 H=2610:00 +CPU= 42.238
  13:30:56 STEP 2611 H=2611:00 +CPU= 26.468
  13:31:32 STEP 2612 H=2612:00 +CPU= 28.366
  13:32:22 STEP 2613 H=2613:00 +CPU= 41.596
  13:32:55 STEP 2614 H=2614:00 +CPU= 27.654
  13:33:30 STEP 2615 H=2615:00 +CPU= 28.905
  13:34:29 STEP 2616 H=2616:00 +CPU= 48.878
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_87.zip
Uploading the intermediate file: upload_file_87.zip
  13:35:02 STEP 2617 H=2617:00 +CPU= 27.376
  13:35:36 STEP 2618 H=2618:00 +CPU= 28.560
  13:36:28 STEP 2619 H=2619:00 +CPU= 42.049
  13:37:02 STEP 2620 H=2620:00 +CPU= 28.265
  13:37:36 STEP 2621 H=2621:00 +CPU= 27.904
  13:38:23 STEP 2622 H=2622:00 +CPU= 40.405
  13:38:56 STEP 2623 H=2623:00 +CPU= 26.869
  13:39:30 STEP 2624 H=2624:00 +CPU= 27.548
  13:40:21 STEP 2625 H=2625:00 +CPU= 41.759
  13:40:55 STEP 2626 H=2626:00 +CPU= 27.639
  13:41:29 STEP 2627 H=2627:00 +CPU= 28.203
  13:42:26 STEP 2628 H=2628:00 +CPU= 46.036
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002628
  13:42:59 STEP 2629 H=2629:00 +CPU= 27.942
  13:43:35 STEP 2630 H=2630:00 +CPU= 29.219
  13:44:25 STEP 2631 H=2631:00 +CPU= 42.094
  13:44:59 STEP 2632 H=2632:00 +CPU= 28.546
  13:45:34 STEP 2633 H=2633:00 +CPU= 28.433
  13:46:25 STEP 2634 H=2634:00 +CPU= 41.865
  13:46:58 STEP 2635 H=2635:00 +CPU= 27.570
  13:47:33 STEP 2636 H=2636:00 +CPU= 28.789
  13:48:23 STEP 2637 H=2637:00 +CPU= 41.959
  13:48:57 STEP 2638 H=2638:00 +CPU= 28.099
  13:49:33 STEP 2639 H=2639:00 +CPU= 29.806
  13:50:31 STEP 2640 H=2640:00 +CPU= 47.899
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_88.zip
Uploading the intermediate file: upload_file_88.zip
  13:51:05 STEP 2641 H=2641:00 +CPU= 28.215
  13:51:38 STEP 2642 H=2642:00 +CPU= 28.560
  13:52:27 STEP 2643 H=2643:00 +CPU= 40.906
  13:53:01 STEP 2644 H=2644:00 +CPU= 28.233
  13:53:34 STEP 2645 H=2645:00 +CPU= 27.930
  13:54:22 STEP 2646 H=2646:00 +CPU= 40.209
  13:54:55 STEP 2647 H=2647:00 +CPU= 27.990
  13:55:29 STEP 2648 H=2648:00 +CPU= 28.281
  13:56:18 STEP 2649 H=2649:00 +CPU= 40.781
  13:56:52 STEP 2650 H=2650:00 +CPU= 28.557
  13:57:25 STEP 2651 H=2651:00 +CPU= 27.917
  13:58:19 STEP 2652 H=2652:00 +CPU= 44.302
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002652
  13:58:48 STEP 2653 H=2653:00 +CPU= 25.403
  13:59:18 STEP 2654 H=2654:00 +CPU= 25.654
  14:00:01 STEP 2655 H=2655:00 +CPU= 36.948
  14:00:31 STEP 2656 H=2656:00 +CPU= 26.100
  14:01:00 STEP 2657 H=2657:00 +CPU= 24.700
  14:01:42 STEP 2658 H=2658:00 +CPU= 36.845
  14:02:10 STEP 2659 H=2659:00 +CPU= 24.269
  14:02:41 STEP 2660 H=2660:00 +CPU= 25.160
  14:03:24 STEP 2661 H=2661:00 +CPU= 36.788
  14:03:54 STEP 2662 H=2662:00 +CPU= 25.140
  14:04:26 STEP 2663 H=2663:00 +CPU= 26.201
  14:05:17 STEP 2664 H=2664:00 +CPU= 42.526
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_89.zip
Uploading the intermediate file: upload_file_89.zip
  14:05:47 STEP 2665 H=2665:00 +CPU= 24.690
  14:06:16 STEP 2666 H=2666:00 +CPU= 24.886
  14:07:00 STEP 2667 H=2667:00 +CPU= 36.952
  14:07:29 STEP 2668 H=2668:00 +CPU= 25.491
  14:07:58 STEP 2669 H=2669:00 +CPU= 24.604
  14:08:43 STEP 2670 H=2670:00 +CPU= 37.521
  14:09:12 STEP 2671 H=2671:00 +CPU= 24.683
  14:09:41 STEP 2672 H=2672:00 +CPU= 25.000
  14:10:26 STEP 2673 H=2673:00 +CPU= 36.840
  14:10:56 STEP 2674 H=2674:00 +CPU= 25.128
  14:11:25 STEP 2675 H=2675:00 +CPU= 24.349
  14:12:14 STEP 2676 H=2676:00 +CPU= 41.158
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002676
  14:12:43 STEP 2677 H=2677:00 +CPU= 24.392
  14:13:13 STEP 2678 H=2678:00 +CPU= 25.091
  14:13:56 STEP 2679 H=2679:00 +CPU= 36.834
  14:14:26 STEP 2680 H=2680:00 +CPU= 25.325
  14:14:54 STEP 2681 H=2681:00 +CPU= 24.545
  14:15:35 STEP 2682 H=2682:00 +CPU= 37.146
  14:16:03 STEP 2683 H=2683:00 +CPU= 24.641
  14:16:31 STEP 2684 H=2684:00 +CPU= 25.129
  14:17:12 STEP 2685 H=2685:00 +CPU= 36.785
  14:17:39 STEP 2686 H=2686:00 +CPU= 24.698
  14:18:08 STEP 2687 H=2687:00 +CPU= 26.081
  14:18:56 STEP 2688 H=2688:00 +CPU= 42.582
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_90.zip
Uploading the intermediate file: upload_file_90.zip
  14:19:25 STEP 2689 H=2689:00 +CPU= 24.482
  14:19:52 STEP 2690 H=2690:00 +CPU= 24.862
  14:20:29 STEP 2691 H=2691:00 +CPU= 35.086
  14:20:55 STEP 2692 H=2692:00 +CPU= 24.607
  14:21:20 STEP 2693 H=2693:00 +CPU= 23.965
  14:21:59 STEP 2694 H=2694:00 +CPU= 36.462
  14:22:23 STEP 2695 H=2695:00 +CPU= 23.513
  14:22:50 STEP 2696 H=2696:00 +CPU= 24.927
  14:23:32 STEP 2697 H=2697:00 +CPU= 36.814
  14:24:02 STEP 2698 H=2698:00 +CPU= 24.979
  14:24:30 STEP 2699 H=2699:00 +CPU= 24.709
  14:25:15 STEP 2700 H=2700:00 +CPU= 40.254
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002700
  14:25:44 STEP 2701 H=2701:00 +CPU= 24.724
  14:26:12 STEP 2702 H=2702:00 +CPU= 24.668
  14:26:54 STEP 2703 H=2703:00 +CPU= 36.705
  14:27:23 STEP 2704 H=2704:00 +CPU= 25.044
  14:27:52 STEP 2705 H=2705:00 +CPU= 24.499
  14:28:36 STEP 2706 H=2706:00 +CPU= 37.586
  14:29:06 STEP 2707 H=2707:00 +CPU= 24.778
  14:29:37 STEP 2708 H=2708:00 +CPU= 25.082
  14:30:21 STEP 2709 H=2709:00 +CPU= 36.807
  14:30:51 STEP 2710 H=2710:00 +CPU= 25.230
  14:31:23 STEP 2711 H=2711:00 +CPU= 26.030
  14:32:15 STEP 2712 H=2712:00 +CPU= 42.003
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_91.zip
Uploading the intermediate file: upload_file_91.zip
  14:32:44 STEP 2713 H=2713:00 +CPU= 24.205
  14:33:14 STEP 2714 H=2714:00 +CPU= 24.592
  14:33:59 STEP 2715 H=2715:00 +CPU= 37.015
  14:34:29 STEP 2716 H=2716:00 +CPU= 25.256
  14:34:58 STEP 2717 H=2717:00 +CPU= 24.073
  14:35:44 STEP 2718 H=2718:00 +CPU= 37.629
  14:36:13 STEP 2719 H=2719:00 +CPU= 24.217
  14:36:43 STEP 2720 H=2720:00 +CPU= 24.868
  14:37:28 STEP 2721 H=2721:00 +CPU= 37.069
  14:37:58 STEP 2722 H=2722:00 +CPU= 25.037
  14:38:26 STEP 2723 H=2723:00 +CPU= 23.890
  14:39:07 STEP 2724 H=2724:00 +CPU= 39.010
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002724
  14:39:33 STEP 2725 H=2725:00 +CPU= 23.664
  14:40:00 STEP 2726 H=2726:00 +CPU= 23.830
  14:40:41 STEP 2727 H=2727:00 +CPU= 36.414
  14:41:10 STEP 2728 H=2728:00 +CPU= 24.630
  14:41:38 STEP 2729 H=2729:00 +CPU= 23.345
  14:42:26 STEP 2730 H=2730:00 +CPU= 39.067
  14:42:58 STEP 2731 H=2731:00 +CPU= 25.624
  14:43:32 STEP 2732 H=2732:00 +CPU= 26.394
  14:44:22 STEP 2733 H=2733:00 +CPU= 39.934
  14:44:56 STEP 2734 H=2734:00 +CPU= 26.615
  14:45:29 STEP 2735 H=2735:00 +CPU= 26.416
  14:46:28 STEP 2736 H=2736:00 +CPU= 46.476
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_92.zip
Uploading the intermediate file: upload_file_92.zip
Uploading trickle at timestep: 9849600
  14:47:02 STEP 2737 H=2737:00 +CPU= 26.135
  14:47:35 STEP 2738 H=2738:00 +CPU= 26.225
  14:48:21 STEP 2739 H=2739:00 +CPU= 37.698
  14:48:51 STEP 2740 H=2740:00 +CPU= 24.732
  14:49:19 STEP 2741 H=2741:00 +CPU= 23.277
  14:50:05 STEP 2742 H=2742:00 +CPU= 38.031
  14:50:34 STEP 2743 H=2743:00 +CPU= 23.538
  14:51:03 STEP 2744 H=2744:00 +CPU= 24.087
  14:51:49 STEP 2745 H=2745:00 +CPU= 37.602
  14:52:19 STEP 2746 H=2746:00 +CPU= 24.644
  14:52:47 STEP 2747 H=2747:00 +CPU= 23.766
  14:53:41 STEP 2748 H=2748:00 +CPU= 41.656
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002748
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0023
(argv4) batchid: 958
(argv5) wuid: 12174667
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/21
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7db76cc0ef87c52e0949750383e2c0fb to: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip
ic_ancil_file: ic_ancil_12174667
ifsdata_file: ifsdata_12174667
climate_data_file: clim_data_12174667
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7d56b2d96fd43574459082e0fe824891 to: /var/lib/boinc/slots/21/ic_ancil_12174667.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/21/ic_ancil_12174667.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3f05a2cabd27cda893615872183ddf12 to: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip
Checking for progress XML file: /var/lib/boinc/slots/21/progress_file_12174667.xml
Opened progress file ok : /var/lib/boinc/slots/21/progress_file_12174667.xml
-- Model is restarting --
Adjusting last_iter, 2749, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/21/progress_file_12174667.xml
last_cpu_time: 64619
upload_file_number: 93
last_iter: 2735
last_upload: 9849600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911
The child process has been launched with process id: 3784
Executing the command: /var/lib/boinc/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:38:57 STEP 2736 H=2736:00 +CPU= 38.864
  18:39:22 STEP 2737 H=2737:00 +CPU= 23.793
  18:39:49 STEP 2738 H=2738:00 +CPU= 24.987
  18:40:25 STEP 2739 H=2739:00 +CPU= 34.536
  18:40:52 STEP 2740 H=2740:00 +CPU= 24.925
  18:41:17 STEP 2741 H=2741:00 +CPU= 23.780
  18:41:54 STEP 2742 H=2742:00 +CPU= 34.368
  18:42:19 STEP 2743 H=2743:00 +CPU= 24.160
  18:42:45 STEP 2744 H=2744:00 +CPU= 24.569
  18:43:22 STEP 2745 H=2745:00 +CPU= 34.559
  18:43:48 STEP 2746 H=2746:00 +CPU= 24.824
  18:44:13 STEP 2747 H=2747:00 +CPU= 23.654
  18:44:54 STEP 2748 H=2748:00 +CPU= 38.573
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002748
  18:45:20 STEP 2749 H=2749:00 +CPU= 24.599
  18:45:46 STEP 2750 H=2750:00 +CPU= 25.032
  18:46:21 STEP 2751 H=2751:00 +CPU= 33.317
  18:46:47 STEP 2752 H=2752:00 +CPU= 24.757
  18:47:12 STEP 2753 H=2753:00 +CPU= 23.933
  18:47:49 STEP 2754 H=2754:00 +CPU= 34.599
  18:48:14 STEP 2755 H=2755:00 +CPU= 24.005
  18:48:40 STEP 2756 H=2756:00 +CPU= 24.065
  18:49:15 STEP 2757 H=2757:00 +CPU= 33.859
  18:49:41 STEP 2758 H=2758:00 +CPU= 24.362
  18:50:07 STEP 2759 H=2759:00 +CPU= 24.864
  18:50:50 STEP 2760 H=2760:00 +CPU= 40.557
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_93.zip
Uploading the intermediate file: upload_file_93.zip
  18:51:16 STEP 2761 H=2761:00 +CPU= 24.196
  18:51:42 STEP 2762 H=2762:00 +CPU= 23.963
  18:52:12 STEP 2763 H=2763:00 +CPU= 29.198
  18:52:28 STEP 2764 H=2764:00 +CPU= 15.549
  18:52:44 STEP 2765 H=2765:00 +CPU= 15.235
  18:53:11 STEP 2766 H=2766:00 +CPU= 25.961
  18:53:29 STEP 2767 H=2767:00 +CPU= 17.364
  18:53:47 STEP 2768 H=2768:00 +CPU= 17.682
  18:54:14 STEP 2769 H=2769:00 +CPU= 25.740
  18:54:33 STEP 2770 H=2770:00 +CPU= 18.133
  18:54:51 STEP 2771 H=2771:00 +CPU= 17.403
  18:55:20 STEP 2772 H=2772:00 +CPU= 28.260
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002772
  18:55:38 STEP 2773 H=2773:00 +CPU= 17.340
  18:55:56 STEP 2774 H=2774:00 +CPU= 17.594
  18:56:23 STEP 2775 H=2775:00 +CPU= 25.686
  18:56:41 STEP 2776 H=2776:00 +CPU= 17.757
  18:56:59 STEP 2777 H=2777:00 +CPU= 17.230
  18:57:26 STEP 2778 H=2778:00 +CPU= 26.181
  18:57:44 STEP 2779 H=2779:00 +CPU= 17.171
  18:58:02 STEP 2780 H=2780:00 +CPU= 17.690
  18:58:29 STEP 2781 H=2781:00 +CPU= 25.767
  18:58:48 STEP 2782 H=2782:00 +CPU= 17.997
  18:59:07 STEP 2783 H=2783:00 +CPU= 18.159
  18:59:39 STEP 2784 H=2784:00 +CPU= 30.389
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_94.zip
Uploading the intermediate file: upload_file_94.zip
  18:59:56 STEP 2785 H=2785:00 +CPU= 16.941
  19:00:15 STEP 2786 H=2786:00 +CPU= 17.899
  19:00:42 STEP 2787 H=2787:00 +CPU= 26.040
  19:01:02 STEP 2788 H=2788:00 +CPU= 18.374
  19:01:20 STEP 2789 H=2789:00 +CPU= 17.504
  19:01:47 STEP 2790 H=2790:00 +CPU= 26.541
  19:02:06 STEP 2791 H=2791:00 +CPU= 17.712
  19:02:24 STEP 2792 H=2792:00 +CPU= 17.402
  19:02:51 STEP 2793 H=2793:00 +CPU= 25.621
  19:03:09 STEP 2794 H=2794:00 +CPU= 17.593
  19:03:27 STEP 2795 H=2795:00 +CPU= 17.290
  19:03:57 STEP 2796 H=2796:00 +CPU= 28.872
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002796
  19:04:15 STEP 2797 H=2797:00 +CPU= 17.594
  19:04:34 STEP 2798 H=2798:00 +CPU= 17.729
  19:05:01 STEP 2799 H=2799:00 +CPU= 25.866
  19:05:19 STEP 2800 H=2800:00 +CPU= 17.699
  19:05:37 STEP 2801 H=2801:00 +CPU= 16.941
  19:06:05 STEP 2802 H=2802:00 +CPU= 26.693
  19:06:23 STEP 2803 H=2803:00 +CPU= 17.268
  19:06:41 STEP 2804 H=2804:00 +CPU= 17.571
  19:07:08 STEP 2805 H=2805:00 +CPU= 26.049
  19:07:27 STEP 2806 H=2806:00 +CPU= 17.924
  19:07:45 STEP 2807 H=2807:00 +CPU= 18.020
  19:08:17 STEP 2808 H=2808:00 +CPU= 30.505
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_95.zip
Uploading the intermediate file: upload_file_95.zip
  19:08:36 STEP 2809 H=2809:00 +CPU= 17.297
  19:08:54 STEP 2810 H=2810:00 +CPU= 18.046
  19:09:21 STEP 2811 H=2811:00 +CPU= 25.971
  19:09:40 STEP 2812 H=2812:00 +CPU= 18.266
  19:09:59 STEP 2813 H=2813:00 +CPU= 17.484
  19:10:26 STEP 2814 H=2814:00 +CPU= 26.765
  19:10:45 STEP 2815 H=2815:00 +CPU= 17.518
  19:11:03 STEP 2816 H=2816:00 +CPU= 17.428
  19:11:30 STEP 2817 H=2817:00 +CPU= 26.300
  19:11:49 STEP 2818 H=2818:00 +CPU= 18.106
  19:12:07 STEP 2819 H=2819:00 +CPU= 17.345
  19:12:37 STEP 2820 H=2820:00 +CPU= 29.089
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002820
  19:12:55 STEP 2821 H=2821:00 +CPU= 17.263
  19:13:14 STEP 2822 H=2822:00 +CPU= 17.487
  19:13:41 STEP 2823 H=2823:00 +CPU= 26.040
  19:13:59 STEP 2824 H=2824:00 +CPU= 18.252
  19:14:17 STEP 2825 H=2825:00 +CPU= 17.224
  19:14:45 STEP 2826 H=2826:00 +CPU= 26.693
  19:15:03 STEP 2827 H=2827:00 +CPU= 17.532
  19:15:22 STEP 2828 H=2828:00 +CPU= 17.815
  19:15:49 STEP 2829 H=2829:00 +CPU= 25.845
  19:16:07 STEP 2830 H=2830:00 +CPU= 17.907
  19:16:26 STEP 2831 H=2831:00 +CPU= 18.088
  19:16:58 STEP 2832 H=2832:00 +CPU= 30.580
Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_96.zip
Uploading the intermediate file: upload_file_96.zip
  19:17:16 STEP 2833 H=2833:00 +CPU= 17.132
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:50:44 1531398 22259336 oifs_43r3_ps_0023_1989050100_123_958_12174667_0 9,849,600 64,260 0.0065
09 May 2023 15:48:29 1531398 22259336 oifs_43r3_ps_0023_1989050100_123_958_12174667_0 8,294,400 50,659 0.0061
09 May 2023 15:47:46 1531398 22259336 oifs_43r3_ps_0023_1989050100_123_958_12174667_0 6,220,800 32,584 0.0052
09 May 2023 15:47:35 1531398 22259336 oifs_43r3_ps_0023_1989050100_123_958_12174667_0 5,356,800 26,241 0.0049
09 May 2023 15:47:24 1531398 22259336 oifs_43r3_ps_0023_1989050100_123_958_12174667_0 4,492,800 19,902 0.0044
09 May 2023 15:47:13 1531398 22259336 oifs_43r3_ps_0023_1989050100_123_958_12174667_0 3,628,800 13,576 0.0037
09 May 2023 15:47:07 1531398 22259336 oifs_43r3_ps_0023_1989050100_123_958_12174667_0 2,764,800 7,198 0.0026
09 May 2023 15:46:35 1531398 22259336 oifs_43r3_ps_0023_1989050100_123_958_12174667_0 1,728,000 12,020 0.0070
09 May 2023 15:46:29 1531398 22259336 oifs_43r3_ps_0023_1989050100_123_958_12174667_0 864,000 6,605 0.0076


©2024 climateprediction.net