climateprediction.net home page
Task 22253427

Task 22253427

Name oifs_43r3_ps_0503_1983050100_123_952_12169147_0
Workunit 12169147
Created 21 Dec 2022, 0:56:55 UTC
Sent 21 Dec 2022, 17:10:47 UTC
Report deadline 20 Jan 2023, 17:10:47 UTC
Received 24 Dec 2022, 1:28:53 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1531398
Run time 1 days 0 hours 2 min 43 sec
CPU time 18 hours 48 min 19 sec
Validate state Invalid
Credit 13,910.40
Device peak FLOPS 7.99 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,792.14 MB
Peak swap size 4,978.10 MB
Peak disk usage 7,172.04 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
20/ifsdata/ifsdata_12169147.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/20/ifsdata/ifsdata_12169147.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/20/159l_2/clim_data_12169147.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/20/159l_2/clim_data_12169147.zip
Checking for progress XML file: /var/lib/boinc/slots/20/progress_file_12169147.xml
Opened progress file ok : /var/lib/boinc/slots/20/progress_file_12169147.xml
-- Model is restarting --
Adjusting last_iter, 2521, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/20/progress_file_12169147.xml
last_cpu_time: 57250
upload_file_number: 84
last_iter: 2519
last_upload: 9072000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840
The child process has been launched with process id: 6328
Executing the command: /var/lib/boinc/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6328:6328] [20221222:232659:1671751619:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1983050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0503
(argv4) batchid: 952
(argv5) wuid: 12169147
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/20
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fbfaf8c01e1337ddaafbefc5d9d64ca2 to: /var/lib/boinc/slots/20/oifs_43r3_ps_0503_1983050100_123_952_12169147.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/20/oifs_43r3_ps_0503_1983050100_123_952_12169147.zip
ic_ancil_file: ic_ancil_12169147
ifsdata_file: ifsdata_12169147
climate_data_file: clim_data_12169147
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4b2cde83478577de421a2afd6aa88b8a to: /var/lib/boinc/slots/20/ic_ancil_12169147.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/20/ic_ancil_12169147.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5a4f8240b4722bda7a29f53ad9f0d870 to: /var/lib/boinc/slots/20/ifsdata/ifsdata_12169147.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/20/ifsdata/ifsdata_12169147.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/20/159l_2/clim_data_12169147.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/20/159l_2/clim_data_12169147.zip
Checking for progress XML file: /var/lib/boinc/slots/20/progress_file_12169147.xml
Opened progress file ok : /var/lib/boinc/slots/20/progress_file_12169147.xml
-- Model is restarting --
Adjusting last_iter, 2520, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/20/progress_file_12169147.xml
last_cpu_time: 57258
upload_file_number: 84
last_iter: 2519
last_upload: 9072000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840
The child process has been launched with process id: 7015
Executing the command: /var/lib/boinc/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7015:7015] [20221223:122632:1671798392:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:27:19 STEP 2520 H=2520:00 +CPU= 42.328
  12:27:43 STEP 2521 H=2521:00 +CPU= 23.618
  12:28:03 STEP 2522 H=2522:00 +CPU= 19.472
  12:28:33 STEP 2523 H=2523:00 +CPU= 29.852
  12:28:56 STEP 2524 H=2524:00 +CPU= 20.533
  12:29:18 STEP 2525 H=2525:00 +CPU= 20.392
  12:29:49 STEP 2526 H=2526:00 +CPU= 30.641
  12:30:10 STEP 2527 H=2527:00 +CPU= 20.871
  12:30:32 STEP 2528 H=2528:00 +CPU= 20.614
  12:31:02 STEP 2529 H=2529:00 +CPU= 29.953
  12:31:25 STEP 2530 H=2530:00 +CPU= 21.765
  12:31:48 STEP 2531 H=2531:00 +CPU= 21.506
  12:32:34 STEP 2532 H=2532:00 +CPU= 39.793
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002532
  12:33:02 STEP 2533 H=2533:00 +CPU= 23.792
  12:33:34 STEP 2534 H=2534:00 +CPU= 25.454
  12:34:15 STEP 2535 H=2535:00 +CPU= 35.602
  12:34:44 STEP 2536 H=2536:00 +CPU= 25.487
  12:35:12 STEP 2537 H=2537:00 +CPU= 23.827
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:36:08 STEP 2538 H=2538:00 +CPU= 34.939
  12:36:41 STEP 2539 H=2539:00 +CPU= 24.122
  12:37:14 STEP 2540 H=2540:00 +CPU= 24.639
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:38:33 STEP 2541 H=2541:00 +CPU= 34.092
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:39:17 STEP 2542 H=2542:00 +CPU= 24.310
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:40:01 STEP 2543 H=2543:00 +CPU= 26.289
  12:40:48 STEP 2544 H=2544:00 +CPU= 41.276
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_84.zip
Uploading the intermediate file: upload_file_84.zip
  12:41:14 STEP 2545 H=2545:00 +CPU= 22.157
  12:41:44 STEP 2546 H=2546:00 +CPU= 23.869
  12:42:24 STEP 2547 H=2547:00 +CPU= 33.303
  12:42:53 STEP 2548 H=2548:00 +CPU= 23.861
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:43:34 STEP 2549 H=2549:00 +CPU= 22.447
  12:44:17 STEP 2550 H=2550:00 +CPU= 34.912
  12:44:46 STEP 2551 H=2551:00 +CPU= 23.223
  12:45:15 STEP 2552 H=2552:00 +CPU= 24.021
  12:45:56 STEP 2553 H=2553:00 +CPU= 33.886
  12:46:26 STEP 2554 H=2554:00 +CPU= 23.970
  12:46:53 STEP 2555 H=2555:00 +CPU= 22.447
  12:47:30 STEP 2556 H=2556:00 +CPU= 32.542
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002556
  12:47:54 STEP 2557 H=2557:00 +CPU= 19.617
  12:48:21 STEP 2558 H=2558:00 +CPU= 21.224
  12:48:56 STEP 2559 H=2559:00 +CPU= 29.057
  12:49:20 STEP 2560 H=2560:00 +CPU= 19.886
  12:49:43 STEP 2561 H=2561:00 +CPU= 19.407
  12:50:23 STEP 2562 H=2562:00 +CPU= 30.748
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:51:16 STEP 2563 H=2563:00 +CPU= 22.367
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:51:55 STEP 2564 H=2564:00 +CPU= 23.051
  12:52:36 STEP 2565 H=2565:00 +CPU= 34.232
  12:53:06 STEP 2566 H=2566:00 +CPU= 23.910
  12:53:36 STEP 2567 H=2567:00 +CPU= 24.464
  12:54:27 STEP 2568 H=2568:00 +CPU= 40.725
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_85.zip
Uploading the intermediate file: upload_file_85.zip
  12:54:57 STEP 2569 H=2569:00 +CPU= 23.626
  12:55:25 STEP 2570 H=2570:00 +CPU= 23.462
  12:56:07 STEP 2571 H=2571:00 +CPU= 34.292
  12:56:36 STEP 2572 H=2572:00 +CPU= 23.624
  12:57:04 STEP 2573 H=2573:00 +CPU= 23.233
  12:57:46 STEP 2574 H=2574:00 +CPU= 34.936
  12:58:14 STEP 2575 H=2575:00 +CPU= 22.996
  12:58:43 STEP 2576 H=2576:00 +CPU= 23.332
  12:59:22 STEP 2577 H=2577:00 +CPU= 33.579
  12:59:48 STEP 2578 H=2578:00 +CPU= 21.727
  13:00:12 STEP 2579 H=2579:00 +CPU= 20.276
  13:00:56 STEP 2580 H=2580:00 +CPU= 36.044
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002580
  13:01:23 STEP 2581 H=2581:00 +CPU= 22.158
  13:01:46 STEP 2582 H=2582:00 +CPU= 20.562
  13:02:22 STEP 2583 H=2583:00 +CPU= 30.960
  13:02:48 STEP 2584 H=2584:00 +CPU= 22.330
  13:03:10 STEP 2585 H=2585:00 +CPU= 19.814
  13:03:48 STEP 2586 H=2586:00 +CPU= 32.354
  13:04:15 STEP 2587 H=2587:00 +CPU= 22.488
  13:04:43 STEP 2588 H=2588:00 +CPU= 22.762
  13:05:22 STEP 2589 H=2589:00 +CPU= 33.298
  13:05:50 STEP 2590 H=2590:00 +CPU= 22.963
  13:06:17 STEP 2591 H=2591:00 +CPU= 22.529
  13:07:04 STEP 2592 H=2592:00 +CPU= 39.096
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_86.zip
Uploading the intermediate file: upload_file_86.zip
  13:07:31 STEP 2593 H=2593:00 +CPU= 22.373
  13:07:59 STEP 2594 H=2594:00 +CPU= 22.447
  13:08:39 STEP 2595 H=2595:00 +CPU= 33.640
  13:09:06 STEP 2596 H=2596:00 +CPU= 22.458
  13:09:32 STEP 2597 H=2597:00 +CPU= 21.781
  13:10:10 STEP 2598 H=2598:00 +CPU= 32.371
  13:10:33 STEP 2599 H=2599:00 +CPU= 20.309
  13:10:58 STEP 2600 H=2600:00 +CPU= 21.386
  13:11:35 STEP 2601 H=2601:00 +CPU= 31.836
  13:12:02 STEP 2602 H=2602:00 +CPU= 22.266
  13:12:28 STEP 2603 H=2603:00 +CPU= 22.106
  13:13:12 STEP 2604 H=2604:00 +CPU= 36.803
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002604
  13:13:37 STEP 2605 H=2605:00 +CPU= 21.831
  13:14:03 STEP 2606 H=2606:00 +CPU= 22.216
  13:14:41 STEP 2607 H=2607:00 +CPU= 33.018
  13:15:09 STEP 2608 H=2608:00 +CPU= 23.901
  13:15:36 STEP 2609 H=2609:00 +CPU= 22.935
  13:16:15 STEP 2610 H=2610:00 +CPU= 35.112
  13:16:43 STEP 2611 H=2611:00 +CPU= 23.280
  13:17:09 STEP 2612 H=2612:00 +CPU= 23.025
  13:17:49 STEP 2613 H=2613:00 +CPU= 34.388
  13:18:17 STEP 2614 H=2614:00 +CPU= 23.540
  13:18:46 STEP 2615 H=2615:00 +CPU= 24.078
  13:19:32 STEP 2616 H=2616:00 +CPU= 40.275
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_87.zip
Uploading the intermediate file: upload_file_87.zip
  13:20:00 STEP 2617 H=2617:00 +CPU= 23.781
  13:20:29 STEP 2618 H=2618:00 +CPU= 24.147
  13:21:13 STEP 2619 H=2619:00 +CPU= 36.192
  13:21:43 STEP 2620 H=2620:00 +CPU= 24.288
  13:22:09 STEP 2621 H=2621:00 +CPU= 22.674
  13:22:52 STEP 2622 H=2622:00 +CPU= 36.479
  13:23:21 STEP 2623 H=2623:00 +CPU= 23.671
  13:23:50 STEP 2624 H=2624:00 +CPU= 24.732
  13:24:32 STEP 2625 H=2625:00 +CPU= 36.012
  13:25:03 STEP 2626 H=2626:00 +CPU= 25.552
  13:25:33 STEP 2627 H=2627:00 +CPU= 25.066
  13:26:21 STEP 2628 H=2628:00 +CPU= 41.791
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002628
  13:26:53 STEP 2629 H=2629:00 +CPU= 25.797
  13:27:27 STEP 2630 H=2630:00 +CPU= 26.581
  13:28:19 STEP 2631 H=2631:00 +CPU= 41.351
  13:28:52 STEP 2632 H=2632:00 +CPU= 27.156
  13:29:27 STEP 2633 H=2633:00 +CPU= 27.769
  13:30:19 STEP 2634 H=2634:00 +CPU= 42.526
  13:30:51 STEP 2635 H=2635:00 +CPU= 26.436
  13:31:25 STEP 2636 H=2636:00 +CPU= 27.692
  13:32:16 STEP 2637 H=2637:00 +CPU= 41.539
  13:32:49 STEP 2638 H=2638:00 +CPU= 27.689
  13:33:25 STEP 2639 H=2639:00 +CPU= 28.622
  13:34:22 STEP 2640 H=2640:00 +CPU= 48.483
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_88.zip
Uploading the intermediate file: upload_file_88.zip
  13:34:54 STEP 2641 H=2641:00 +CPU= 26.443
  13:35:29 STEP 2642 H=2642:00 +CPU= 28.592
  13:36:19 STEP 2643 H=2643:00 +CPU= 41.769
  13:36:52 STEP 2644 H=2644:00 +CPU= 27.307
  13:37:25 STEP 2645 H=2645:00 +CPU= 27.815
  13:38:13 STEP 2646 H=2646:00 +CPU= 40.583
  13:38:44 STEP 2647 H=2647:00 +CPU= 25.812
  13:39:18 STEP 2648 H=2648:00 +CPU= 27.702
  13:40:08 STEP 2649 H=2649:00 +CPU= 40.254
  13:40:42 STEP 2650 H=2650:00 +CPU= 27.009
  13:41:16 STEP 2651 H=2651:00 +CPU= 27.858
  13:42:13 STEP 2652 H=2652:00 +CPU= 45.950
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002652
  13:42:45 STEP 2653 H=2653:00 +CPU= 26.733
  13:43:20 STEP 2654 H=2654:00 +CPU= 28.913
  13:44:11 STEP 2655 H=2655:00 +CPU= 41.579
  13:44:43 STEP 2656 H=2656:00 +CPU= 27.381
  13:45:18 STEP 2657 H=2657:00 +CPU= 28.599
  13:46:09 STEP 2658 H=2658:00 +CPU= 41.957
  13:46:40 STEP 2659 H=2659:00 +CPU= 26.074
  13:47:14 STEP 2660 H=2660:00 +CPU= 28.356
  13:48:04 STEP 2661 H=2661:00 +CPU= 41.285
  13:48:37 STEP 2662 H=2662:00 +CPU= 27.527
  13:49:11 STEP 2663 H=2663:00 +CPU= 29.308
  13:50:10 STEP 2664 H=2664:00 +CPU= 48.347
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_89.zip
Uploading the intermediate file: upload_file_89.zip
  13:50:42 STEP 2665 H=2665:00 +CPU= 25.811
  13:51:17 STEP 2666 H=2666:00 +CPU= 28.774
  13:52:07 STEP 2667 H=2667:00 +CPU= 41.052
  13:52:39 STEP 2668 H=2668:00 +CPU= 26.860
  13:53:13 STEP 2669 H=2669:00 +CPU= 27.665
  13:54:01 STEP 2670 H=2670:00 +CPU= 40.910
  13:54:32 STEP 2671 H=2671:00 +CPU= 26.115
  13:55:07 STEP 2672 H=2672:00 +CPU= 28.277
  13:55:57 STEP 2673 H=2673:00 +CPU= 40.997
  13:56:31 STEP 2674 H=2674:00 +CPU= 27.780
  13:57:04 STEP 2675 H=2675:00 +CPU= 27.677
  13:58:00 STEP 2676 H=2676:00 +CPU= 45.892
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002676
  13:58:29 STEP 2677 H=2677:00 +CPU= 24.484
  13:58:59 STEP 2678 H=2678:00 +CPU= 25.883
  13:59:42 STEP 2679 H=2679:00 +CPU= 37.242
  14:00:11 STEP 2680 H=2680:00 +CPU= 24.815
  14:00:40 STEP 2681 H=2681:00 +CPU= 25.170
  14:01:24 STEP 2682 H=2682:00 +CPU= 37.746
  14:01:51 STEP 2683 H=2683:00 +CPU= 23.500
  14:02:21 STEP 2684 H=2684:00 +CPU= 25.624
  14:03:04 STEP 2685 H=2685:00 +CPU= 36.744
  14:03:33 STEP 2686 H=2686:00 +CPU= 24.601
  14:04:04 STEP 2687 H=2687:00 +CPU= 26.158
  14:04:55 STEP 2688 H=2688:00 +CPU= 42.540
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_90.zip
Uploading the intermediate file: upload_file_90.zip
  14:05:24 STEP 2689 H=2689:00 +CPU= 23.600
  14:05:54 STEP 2690 H=2690:00 +CPU= 25.447
  14:06:37 STEP 2691 H=2691:00 +CPU= 36.498
  14:07:05 STEP 2692 H=2692:00 +CPU= 24.375
  14:07:34 STEP 2693 H=2693:00 +CPU= 24.803
  14:08:18 STEP 2694 H=2694:00 +CPU= 37.482
  14:08:46 STEP 2695 H=2695:00 +CPU= 24.102
  14:09:15 STEP 2696 H=2696:00 +CPU= 24.830
  14:09:59 STEP 2697 H=2697:00 +CPU= 36.604
  14:10:28 STEP 2698 H=2698:00 +CPU= 24.643
  14:10:57 STEP 2699 H=2699:00 +CPU= 24.432
  14:11:46 STEP 2700 H=2700:00 +CPU= 40.888
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002700
  14:12:14 STEP 2701 H=2701:00 +CPU= 23.898
  14:12:45 STEP 2702 H=2702:00 +CPU= 25.011
  14:13:29 STEP 2703 H=2703:00 +CPU= 37.214
  14:13:57 STEP 2704 H=2704:00 +CPU= 24.095
  14:14:26 STEP 2705 H=2705:00 +CPU= 24.414
  14:15:09 STEP 2706 H=2706:00 +CPU= 37.598
  14:15:34 STEP 2707 H=2707:00 +CPU= 23.424
  14:16:02 STEP 2708 H=2708:00 +CPU= 24.869
  14:16:43 STEP 2709 H=2709:00 +CPU= 36.439
  14:17:11 STEP 2710 H=2710:00 +CPU= 24.625
  14:17:38 STEP 2711 H=2711:00 +CPU= 24.625
  14:18:26 STEP 2712 H=2712:00 +CPU= 43.016
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_91.zip
Uploading the intermediate file: upload_file_91.zip
  14:18:53 STEP 2713 H=2713:00 +CPU= 23.595
  14:19:22 STEP 2714 H=2714:00 +CPU= 24.975
  14:20:02 STEP 2715 H=2715:00 +CPU= 36.258
  14:20:27 STEP 2716 H=2716:00 +CPU= 23.537
  14:20:52 STEP 2717 H=2717:00 +CPU= 23.729
  14:21:30 STEP 2718 H=2718:00 +CPU= 36.328
  14:21:55 STEP 2719 H=2719:00 +CPU= 23.701
  14:22:20 STEP 2720 H=2720:00 +CPU= 24.202
  14:23:01 STEP 2721 H=2721:00 +CPU= 37.016
  14:23:29 STEP 2722 H=2722:00 +CPU= 24.878
  14:23:57 STEP 2723 H=2723:00 +CPU= 24.017
  14:24:43 STEP 2724 H=2724:00 +CPU= 40.947
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002724
  14:25:10 STEP 2725 H=2725:00 +CPU= 24.139
  14:25:40 STEP 2726 H=2726:00 +CPU= 25.294
  14:26:21 STEP 2727 H=2727:00 +CPU= 36.256
  14:26:49 STEP 2728 H=2728:00 +CPU= 24.593
  14:27:17 STEP 2729 H=2729:00 +CPU= 24.013
  14:28:01 STEP 2730 H=2730:00 +CPU= 37.251
  14:28:29 STEP 2731 H=2731:00 +CPU= 24.256
  14:28:59 STEP 2732 H=2732:00 +CPU= 24.917
  14:29:45 STEP 2733 H=2733:00 +CPU= 37.196
  14:30:14 STEP 2734 H=2734:00 +CPU= 24.554
  14:30:44 STEP 2735 H=2735:00 +CPU= 25.075
  14:31:37 STEP 2736 H=2736:00 +CPU= 42.774
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_92.zip
Uploading the intermediate file: upload_file_92.zip
  14:32:07 STEP 2737 H=2737:00 +CPU= 24.123
  14:32:37 STEP 2738 H=2738:00 +CPU= 24.709
  14:33:22 STEP 2739 H=2739:00 +CPU= 36.558
  14:33:53 STEP 2740 H=2740:00 +CPU= 24.801
  14:34:22 STEP 2741 H=2741:00 +CPU= 24.267
  14:35:08 STEP 2742 H=2742:00 +CPU= 37.363
  14:35:38 STEP 2743 H=2743:00 +CPU= 24.526
  14:36:09 STEP 2744 H=2744:00 +CPU= 24.974
  14:36:54 STEP 2745 H=2745:00 +CPU= 36.992
  14:37:25 STEP 2746 H=2746:00 +CPU= 25.074
  14:37:53 STEP 2747 H=2747:00 +CPU= 23.942
  14:38:39 STEP 2748 H=2748:00 +CPU= 39.947
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002748
  14:39:03 STEP 2749 H=2749:00 +CPU= 22.903
  14:39:28 STEP 2750 H=2750:00 +CPU= 23.764
  14:40:08 STEP 2751 H=2751:00 +CPU= 35.962
  14:40:36 STEP 2752 H=2752:00 +CPU= 24.447
  14:41:04 STEP 2753 H=2753:00 +CPU= 23.897
  14:41:48 STEP 2754 H=2754:00 +CPU= 37.104
  14:42:20 STEP 2755 H=2755:00 +CPU= 25.355
  14:42:54 STEP 2756 H=2756:00 +CPU= 26.926
  14:43:44 STEP 2757 H=2757:00 +CPU= 39.510
  14:44:18 STEP 2758 H=2758:00 +CPU= 26.635
  14:44:52 STEP 2759 H=2759:00 +CPU= 27.383
  14:45:48 STEP 2760 H=2760:00 +CPU= 44.644
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_93.zip
Uploading the intermediate file: upload_file_93.zip
Uploading trickle at timestep: 9936000
  14:46:22 STEP 2761 H=2761:00 +CPU= 26.332
  14:46:56 STEP 2762 H=2762:00 +CPU= 26.822
  14:47:46 STEP 2763 H=2763:00 +CPU= 39.465
  14:48:16 STEP 2764 H=2764:00 +CPU= 24.516
  14:48:45 STEP 2765 H=2765:00 +CPU= 24.325
  14:49:32 STEP 2766 H=2766:00 +CPU= 37.999
  14:50:01 STEP 2767 H=2767:00 +CPU= 24.378
  14:50:31 STEP 2768 H=2768:00 +CPU= 24.705
  14:51:16 STEP 2769 H=2769:00 +CPU= 37.139
  14:51:47 STEP 2770 H=2770:00 +CPU= 24.996
  14:52:15 STEP 2771 H=2771:00 +CPU= 24.072
  14:53:05 STEP 2772 H=2772:00 +CPU= 41.133
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002772
  14:53:35 STEP 2773 H=2773:00 +CPU= 23.723
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1983050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0503
(argv4) batchid: 952
(argv5) wuid: 12169147
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/20
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fbfaf8c01e1337ddaafbefc5d9d64ca2 to: /var/lib/boinc/slots/20/oifs_43r3_ps_0503_1983050100_123_952_12169147.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/20/oifs_43r3_ps_0503_1983050100_123_952_12169147.zip
ic_ancil_file: ic_ancil_12169147
ifsdata_file: ifsdata_12169147
climate_data_file: clim_data_12169147
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4b2cde83478577de421a2afd6aa88b8a to: /var/lib/boinc/slots/20/ic_ancil_12169147.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/20/ic_ancil_12169147.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5a4f8240b4722bda7a29f53ad9f0d870 to: /var/lib/boinc/slots/20/ifsdata/ifsdata_12169147.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/20/ifsdata/ifsdata_12169147.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/20/159l_2/clim_data_12169147.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/20/159l_2/clim_data_12169147.zip
Checking for progress XML file: /var/lib/boinc/slots/20/progress_file_12169147.xml
Opened progress file ok : /var/lib/boinc/slots/20/progress_file_12169147.xml
-- Model is restarting --
Adjusting last_iter, 2774, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/20/progress_file_12169147.xml
last_cpu_time: 64596
upload_file_number: 94
last_iter: 2759
last_upload: 9936000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840
The child process has been launched with process id: 3790
Executing the command: /var/lib/boinc/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3790:3790] [20221223:183811:1671820691:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:38:57 STEP 2760 H=2760:00 +CPU= 38.473
  18:39:22 STEP 2761 H=2761:00 +CPU= 23.082
  18:39:48 STEP 2762 H=2762:00 +CPU= 24.541
  18:40:24 STEP 2763 H=2763:00 +CPU= 34.377
  18:40:50 STEP 2764 H=2764:00 +CPU= 24.214
  18:41:15 STEP 2765 H=2765:00 +CPU= 23.311
  18:41:52 STEP 2766 H=2766:00 +CPU= 35.270
  18:42:18 STEP 2767 H=2767:00 +CPU= 24.003
  18:42:43 STEP 2768 H=2768:00 +CPU= 23.911
  18:43:19 STEP 2769 H=2769:00 +CPU= 33.954
  18:43:44 STEP 2770 H=2770:00 +CPU= 24.641
  18:44:09 STEP 2771 H=2771:00 +CPU= 23.656
  18:44:49 STEP 2772 H=2772:00 +CPU= 38.012
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002772
  18:45:14 STEP 2773 H=2773:00 +CPU= 23.551
  18:45:40 STEP 2774 H=2774:00 +CPU= 24.748
  18:46:16 STEP 2775 H=2775:00 +CPU= 34.376
  18:46:41 STEP 2776 H=2776:00 +CPU= 23.884
  18:47:06 STEP 2777 H=2777:00 +CPU= 24.126
  18:47:42 STEP 2778 H=2778:00 +CPU= 33.686
  18:48:07 STEP 2779 H=2779:00 +CPU= 24.157
  18:48:34 STEP 2780 H=2780:00 +CPU= 25.087
  18:49:09 STEP 2781 H=2781:00 +CPU= 33.898
  18:49:35 STEP 2782 H=2782:00 +CPU= 24.643
  18:50:01 STEP 2783 H=2783:00 +CPU= 25.010
  18:50:44 STEP 2784 H=2784:00 +CPU= 39.601
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_94.zip
Uploading the intermediate file: upload_file_94.zip
  18:51:10 STEP 2785 H=2785:00 +CPU= 23.275
  18:51:35 STEP 2786 H=2786:00 +CPU= 24.205
  18:52:08 STEP 2787 H=2787:00 +CPU= 31.441
  18:52:25 STEP 2788 H=2788:00 +CPU= 15.950
  18:52:40 STEP 2789 H=2789:00 +CPU= 14.992
  18:53:07 STEP 2790 H=2790:00 +CPU= 26.335
  18:53:26 STEP 2791 H=2791:00 +CPU= 17.481
  18:53:44 STEP 2792 H=2792:00 +CPU= 17.627
  18:54:11 STEP 2793 H=2793:00 +CPU= 25.901
  18:54:29 STEP 2794 H=2794:00 +CPU= 17.566
  18:54:47 STEP 2795 H=2795:00 +CPU= 17.265
  18:55:16 STEP 2796 H=2796:00 +CPU= 28.098
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002796
  18:55:34 STEP 2797 H=2797:00 +CPU= 17.258
  18:55:53 STEP 2798 H=2798:00 +CPU= 18.213
  18:56:20 STEP 2799 H=2799:00 +CPU= 25.582
  18:56:39 STEP 2800 H=2800:00 +CPU= 17.912
  18:56:57 STEP 2801 H=2801:00 +CPU= 17.583
  18:57:24 STEP 2802 H=2802:00 +CPU= 26.807
  18:57:42 STEP 2803 H=2803:00 +CPU= 17.143
  18:58:00 STEP 2804 H=2804:00 +CPU= 17.441
  18:58:27 STEP 2805 H=2805:00 +CPU= 26.073
  18:58:46 STEP 2806 H=2806:00 +CPU= 17.783
  18:59:05 STEP 2807 H=2807:00 +CPU= 18.220
  18:59:36 STEP 2808 H=2808:00 +CPU= 30.562
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_95.zip
Uploading the intermediate file: upload_file_95.zip
  18:59:55 STEP 2809 H=2809:00 +CPU= 17.119
  19:00:14 STEP 2810 H=2810:00 +CPU= 18.023
  19:00:41 STEP 2811 H=2811:00 +CPU= 26.644
  19:01:00 STEP 2812 H=2812:00 +CPU= 18.019
  19:01:18 STEP 2813 H=2813:00 +CPU= 17.390
  19:01:46 STEP 2814 H=2814:00 +CPU= 26.455
  19:02:04 STEP 2815 H=2815:00 +CPU= 17.495
  19:02:22 STEP 2816 H=2816:00 +CPU= 17.592
  19:02:50 STEP 2817 H=2817:00 +CPU= 26.418
  19:03:08 STEP 2818 H=2818:00 +CPU= 17.775
  19:03:27 STEP 2819 H=2819:00 +CPU= 17.632
  19:03:57 STEP 2820 H=2820:00 +CPU= 28.954
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002820
  19:04:15 STEP 2821 H=2821:00 +CPU= 17.690
  19:04:33 STEP 2822 H=2822:00 +CPU= 17.761
  19:05:00 STEP 2823 H=2823:00 +CPU= 25.686
  19:05:19 STEP 2824 H=2824:00 +CPU= 17.652
  19:05:37 STEP 2825 H=2825:00 +CPU= 17.511
  19:06:05 STEP 2826 H=2826:00 +CPU= 26.612
  19:06:23 STEP 2827 H=2827:00 +CPU= 17.420
  19:06:42 STEP 2828 H=2828:00 +CPU= 17.785
  19:07:08 STEP 2829 H=2829:00 +CPU= 25.994
  19:07:27 STEP 2830 H=2830:00 +CPU= 17.849
  19:07:45 STEP 2831 H=2831:00 +CPU= 17.670
  19:08:17 STEP 2832 H=2832:00 +CPU= 30.665
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_96.zip
Uploading the intermediate file: upload_file_96.zip
  19:08:36 STEP 2833 H=2833:00 +CPU= 17.180
  19:08:54 STEP 2834 H=2834:00 +CPU= 17.782
  19:09:22 STEP 2835 H=2835:00 +CPU= 25.894
  19:09:40 STEP 2836 H=2836:00 +CPU= 17.933
  19:09:58 STEP 2837 H=2837:00 +CPU= 17.371
  19:10:26 STEP 2838 H=2838:00 +CPU= 26.599
  19:10:44 STEP 2839 H=2839:00 +CPU= 17.598
  19:11:03 STEP 2840 H=2840:00 +CPU= 17.978
  19:11:30 STEP 2841 H=2841:00 +CPU= 25.949
  19:11:49 STEP 2842 H=2842:00 +CPU= 18.398
  19:12:07 STEP 2843 H=2843:00 +CPU= 17.265
  19:12:37 STEP 2844 H=2844:00 +CPU= 29.034
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002844
  19:12:55 STEP 2845 H=2845:00 +CPU= 17.242
  19:13:13 STEP 2846 H=2846:00 +CPU= 17.805
  19:13:40 STEP 2847 H=2847:00 +CPU= 26.030
  19:13:59 STEP 2848 H=2848:00 +CPU= 18.001
  19:14:17 STEP 2849 H=2849:00 +CPU= 17.610
  19:14:45 STEP 2850 H=2850:00 +CPU= 26.508
  19:15:03 STEP 2851 H=2851:00 +CPU= 17.395
  19:15:21 STEP 2852 H=2852:00 +CPU= 17.623
  19:15:49 STEP 2853 H=2853:00 +CPU= 26.152
  19:16:07 STEP 2854 H=2854:00 +CPU= 18.213
  19:16:26 STEP 2855 H=2855:00 +CPU= 17.871
  19:16:58 STEP 2856 H=2856:00 +CPU= 30.492
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_97.zip
Uploading the intermediate file: upload_file_97.zip
  19:17:17 STEP 2857 H=2857:00 +CPU= 17.623
  19:17:33 STEP 2858 H=2858:00 +CPU= 15.657
  19:17:57 STEP 2859 H=2859:00 +CPU= 23.172
  19:18:13 STEP 2860 H=2860:00 +CPU= 15.538
  19:18:29 STEP 2861 H=2861:00 +CPU= 14.791
  19:18:54 STEP 2862 H=2862:00 +CPU= 24.009
  19:19:09 STEP 2863 H=2863:00 +CPU= 15.294
  19:19:25 STEP 2864 H=2864:00 +CPU= 15.331
  19:19:49 STEP 2865 H=2865:00 +CPU= 23.118
  19:20:05 STEP 2866 H=2866:00 +CPU= 15.579
  19:20:21 STEP 2867 H=2867:00 +CPU= 15.128
  19:20:47 STEP 2868 H=2868:00 +CPU= 25.761
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002868
  19:21:03 STEP 2869 H=2869:00 +CPU= 15.339
  19:21:19 STEP 2870 H=2870:00 +CPU= 15.643
  19:21:43 STEP 2871 H=2871:00 +CPU= 23.148
  19:21:59 STEP 2872 H=2872:00 +CPU= 15.607
  19:22:15 STEP 2873 H=2873:00 +CPU= 14.949
  19:22:39 STEP 2874 H=2874:00 +CPU= 23.906
  19:22:55 STEP 2875 H=2875:00 +CPU= 14.977
  19:23:11 STEP 2876 H=2876:00 +CPU= 15.384
  19:23:35 STEP 2877 H=2877:00 +CPU= 23.231
  19:23:51 STEP 2878 H=2878:00 +CPU= 15.438
  19:24:07 STEP 2879 H=2879:00 +CPU= 15.514
  19:24:35 STEP 2880 H=2880:00 +CPU= 27.403
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12169147/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0503_1983050100_123_952_12169147_0_r924580840_98.zip
Uploading the intermediate file: upload_file_98.zip
  19:24:51 STEP 2881 H=2881:00 +CPU= 15.023
  19:25:07 STEP 2882 H=2882:00 +CPU= 15.221
  19:25:31 STEP 2883 H=2883:00 +CPU= 23.525
  19:25:47 STEP 2884 H=2884:00 +CPU= 15.451
  19:26:03 STEP 2885 H=2885:00 +CPU= 14.975
  19:26:27 STEP 2886 H=2886:00 +CPU= 23.945
  19:26:43 STEP 2887 H=2887:00 +CPU= 14.927
  19:26:58 STEP 2888 H=2888:00 +CPU= 15.195
  19:27:23 STEP 2889 H=2889:00 +CPU= 23.472
  19:27:39 STEP 2890 H=2890:00 +CPU= 15.557
  19:27:54 STEP 2891 H=2891:00 +CPU= 15.154
  19:28:21 STEP 2892 H=2892:00 +CPU= 25.975
Moving to projects directory: /var/lib/boinc/slots/20/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc/slots/20/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc/slots/20/ICMUAhq0f+002892
  19:28:37 STEP 2893 H=2893:00 +CPU= 15.181
  19:28:53 STEP 2894 H=2894:00 +CPU= 15.378
  19:29:17 STEP 2895 H=2895:00 +CPU= 23.049
  19:29:33 STEP 2896 H=2896:00 +CPU= 15.527
  19:29:48 STEP 2897 H=2897:00 +CPU= 14.772
  19:30:12 STEP 2898 H=2898:00 +CPU= 23.705
  19:30:28 STEP 2899 H=2899:00 +CPU= 15.200
  19:30:44 STEP 2900 H=2900:00 +CPU= 15.127
  19:31:08 STEP 2901 H=2901:00 +CPU= 23.213
  19:31:24 STEP 2902 H=2902:00 +CPU= 15.524
  19:31:40 STEP 2903 H=2903:00 +CPU= 15.660
  19:32:09 STEP 2904 H=2904:00 +CPU= 27.508
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:50:44 1531398 22253427 oifs_43r3_ps_0503_1983050100_123_952_12169147_0 9,936,000 64,205 0.0065
09 May 2023 15:48:29 1531398 22253427 oifs_43r3_ps_0503_1983050100_123_952_12169147_0 8,380,800 50,666 0.0060
09 May 2023 15:47:46 1531398 22253427 oifs_43r3_ps_0503_1983050100_123_952_12169147_0 6,220,800 31,803 0.0051
09 May 2023 15:47:35 1531398 22253427 oifs_43r3_ps_0503_1983050100_123_952_12169147_0 5,356,800 25,655 0.0048
09 May 2023 15:47:24 1531398 22253427 oifs_43r3_ps_0503_1983050100_123_952_12169147_0 4,492,800 19,488 0.0043
09 May 2023 15:47:13 1531398 22253427 oifs_43r3_ps_0503_1983050100_123_952_12169147_0 3,628,800 13,300 0.0037
09 May 2023 15:47:07 1531398 22253427 oifs_43r3_ps_0503_1983050100_123_952_12169147_0 2,764,800 7,072 0.0026
09 May 2023 15:46:35 1531398 22253427 oifs_43r3_ps_0503_1983050100_123_952_12169147_0 1,728,000 11,973 0.0069
09 May 2023 15:46:29 1531398 22253427 oifs_43r3_ps_0503_1983050100_123_952_12169147_0 864,000 6,590 0.0076


©2024 cpdn.org