<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
668cf7829426ffd94699b48b9a2e to: /d6/home/boinc/slots/5/159l_2/clim_data_12203879.zip
Unzipping the climate data zip file: /d6/home/boinc/slots/5/159l_2/clim_data_12203879.zip
Checking for progress XML file: /d6/home/boinc/slots/5/progress_file_12203879.xml
Opened progress file ok : /d6/home/boinc/slots/5/progress_file_12203879.xml
-- Model is restarting --
Adjusting last_iter, 2412, to previous model restart step.
Creating progress file: /d6/home/boinc/slots/5/progress_file_12203879.xml
last_cpu_time: 2591
upload_file_number: 4
last_iter: 2399
last_upload: 8640000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0235_2018050100_123_987_12203879_0_r770809535
The child process has been launched with process id: 11284
Executing the command: /d6/home/boinc/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:11284:11284] [20221230:110111:1672394471:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
11:05:40 STEP 2400 H=2400:00 +CPU= 33.660
11:06:31 STEP 2401 H=2401:00 +CPU= 19.887
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:12:38 STEP 2402 H=2402:00 +CPU= 16.691
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:14:22 STEP 2403 H=2403:00 +CPU= 27.712
Suspend request received from the BOINC client, suspending the child process
11:16:48 STEP 2404 H=2404:00 +CPU= 15.662
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:18:22 STEP 2405 H=2405:00 +CPU= 18.228
11:19:48 STEP 2406 H=2406:00 +CPU= 27.007
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:22:35 STEP 2407 H=2407:00 +CPU= 16.842
11:23:10 STEP 2408 H=2408:00 +CPU= 13.801
Suspend request received from the BOINC client, suspending the child process
11:24:43 STEP 2409 H=2409:00 +CPU= 22.242
Resuming the child process
11:25:30 STEP 2410 H=2410:00 +CPU= 17.537
11:26:03 STEP 2411 H=2411:00 +CPU= 17.344
11:27:13 STEP 2412 H=2412:00 +CPU= 26.280
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002412
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002412
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002412
11:27:47 STEP 2413 H=2413:00 +CPU= 14.088
11:28:19 STEP 2414 H=2414:00 +CPU= 12.482
11:29:11 STEP 2415 H=2415:00 +CPU= 20.317
11:29:56 STEP 2416 H=2416:00 +CPU= 16.835
11:30:45 STEP 2417 H=2417:00 +CPU= 19.990
11:31:49 STEP 2418 H=2418:00 +CPU= 24.775
11:32:36 STEP 2419 H=2419:00 +CPU= 17.244
11:33:23 STEP 2420 H=2420:00 +CPU= 19.620
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:27:07 STEP 2421 H=2421:00 +CPU= 27.798
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:40:57 STEP 2422 H=2422:00 +CPU= 16.874
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0235
(argv4) batchid: 987
(argv5) wuid: 12203879
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /d6/home/boinc/slots/5
Project directory is: /d6/home/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879
..mkdir for temp folder for results failed
Copying: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /d6/home/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /d6/home/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a37947bf475ca8ac2aedb9de3c7e1bfc to: /d6/home/boinc/slots/5/oifs_43r3_ps_0235_2018050100_123_987_12203879.zip
Unzipping the namelist zip file: /d6/home/boinc/slots/5/oifs_43r3_ps_0235_2018050100_123_987_12203879.zip
ic_ancil_file: ic_ancil_12203879
ifsdata_file: ifsdata_12203879
climate_data_file: clim_data_12203879
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5b73891d866a923fd582c819e7820bae to: /d6/home/boinc/slots/5/ic_ancil_12203879.zip
Unzipping the IC ancils zip file: /d6/home/boinc/slots/5/ic_ancil_12203879.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c9c301ab539750247d3396013ffc9bcb to: /d6/home/boinc/slots/5/ifsdata/ifsdata_12203879.zip
Unzipping the ifsdata_zip file: /d6/home/boinc/slots/5/ifsdata/ifsdata_12203879.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /d6/home/boinc/slots/5/159l_2/clim_data_12203879.zip
Unzipping the climate data zip file: /d6/home/boinc/slots/5/159l_2/clim_data_12203879.zip
Checking for progress XML file: /d6/home/boinc/slots/5/progress_file_12203879.xml
Opened progress file ok : /d6/home/boinc/slots/5/progress_file_12203879.xml
-- Model is restarting --
Adjusting last_iter, 2423, to previous model restart step.
Creating progress file: /d6/home/boinc/slots/5/progress_file_12203879.xml
last_cpu_time: 3067
upload_file_number: 4
last_iter: 2399
last_upload: 8640000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0235_2018050100_123_987_12203879_0_r770809535
The child process has been launched with process id: 8595
Executing the command: /d6/home/boinc/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:8595:8595] [20221230:154301:1672411381:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:46:24 STEP 2400 H=2400:00 +CPU= 30.335
Suspend request received from the BOINC client, suspending the child process
15:47:56 STEP 2401 H=2401:00 +CPU= 15.778
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:48:40 STEP 2402 H=2402:00 +CPU= 14.806
15:49:53 STEP 2403 H=2403:00 +CPU= 29.420
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
23:02:44 STEP 2404 H=2404:00 +CPU= 27.433
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
23:07:28 STEP 2405 H=2405:00 +CPU= 11.537
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:16:24 STEP 2406 H=2406:00 +CPU= 19.873
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0235
(argv4) batchid: 987
(argv5) wuid: 12203879
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /d6/home/boinc/slots/5
Project directory is: /d6/home/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879
..mkdir for temp folder for results failed
Copying: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /d6/home/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /d6/home/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a37947bf475ca8ac2aedb9de3c7e1bfc to: /d6/home/boinc/slots/5/oifs_43r3_ps_0235_2018050100_123_987_12203879.zip
Unzipping the namelist zip file: /d6/home/boinc/slots/5/oifs_43r3_ps_0235_2018050100_123_987_12203879.zip
ic_ancil_file: ic_ancil_12203879
ifsdata_file: ifsdata_12203879
climate_data_file: clim_data_12203879
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5b73891d866a923fd582c819e7820bae to: /d6/home/boinc/slots/5/ic_ancil_12203879.zip
Unzipping the IC ancils zip file: /d6/home/boinc/slots/5/ic_ancil_12203879.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c9c301ab539750247d3396013ffc9bcb to: /d6/home/boinc/slots/5/ifsdata/ifsdata_12203879.zip
Unzipping the ifsdata_zip file: /d6/home/boinc/slots/5/ifsdata/ifsdata_12203879.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /d6/home/boinc/slots/5/159l_2/clim_data_12203879.zip
Unzipping the climate data zip file: /d6/home/boinc/slots/5/159l_2/clim_data_12203879.zip
Checking for progress XML file: /d6/home/boinc/slots/5/progress_file_12203879.xml
Opened progress file ok : /d6/home/boinc/slots/5/progress_file_12203879.xml
-- Model is restarting --
Adjusting last_iter, 2407, to previous model restart step.
Creating progress file: /d6/home/boinc/slots/5/progress_file_12203879.xml
last_cpu_time: 3224
upload_file_number: 4
last_iter: 2399
last_upload: 8640000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0235_2018050100_123_987_12203879_0_r770809535
The child process has been launched with process id: 27447
Executing the command: /d6/home/boinc/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:27447:27447] [20221230:231753:1672438673:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
23:18:25 STEP 2400 H=2400:00 +CPU= 20.313
23:18:37 STEP 2401 H=2401:00 +CPU= 11.687
23:18:49 STEP 2402 H=2402:00 +CPU= 11.236
23:19:07 STEP 2403 H=2403:00 +CPU= 17.553
23:19:18 STEP 2404 H=2404:00 +CPU= 11.728
23:19:30 STEP 2405 H=2405:00 +CPU= 11.480
23:19:47 STEP 2406 H=2406:00 +CPU= 17.085
23:19:58 STEP 2407 H=2407:00 +CPU= 10.263
23:20:08 STEP 2408 H=2408:00 +CPU= 9.987
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:20:51 STEP 2409 H=2409:00 +CPU= 16.087
23:21:01 STEP 2410 H=2410:00 +CPU= 9.924
23:21:11 STEP 2411 H=2411:00 +CPU= 9.582
23:21:28 STEP 2412 H=2412:00 +CPU= 17.345
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002412
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002412
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002412
23:21:38 STEP 2413 H=2413:00 +CPU= 9.404
23:21:48 STEP 2414 H=2414:00 +CPU= 10.470
23:22:05 STEP 2415 H=2415:00 +CPU= 16.842
23:22:16 STEP 2416 H=2416:00 +CPU= 10.945
23:22:27 STEP 2417 H=2417:00 +CPU= 10.282
23:22:44 STEP 2418 H=2418:00 +CPU= 17.087
23:22:54 STEP 2419 H=2419:00 +CPU= 10.279
23:23:05 STEP 2420 H=2420:00 +CPU= 10.525
23:23:20 STEP 2421 H=2421:00 +CPU= 15.476
23:23:30 STEP 2422 H=2422:00 +CPU= 9.552
23:23:40 STEP 2423 H=2423:00 +CPU= 9.883
23:23:58 STEP 2424 H=2424:00 +CPU= 17.770
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002424
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002424
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002424
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002400
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002400
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002400
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002412
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002412
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002412
Zipping up the intermediate file: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0235_2018050100_123_987_12203879_0_r770809535_4.zip
Uploading the intermediate file: upload_file_4.zip
23:24:07 STEP 2425 H=2425:00 +CPU= 9.014
23:24:17 STEP 2426 H=2426:00 +CPU= 9.796
23:24:32 STEP 2427 H=2427:00 +CPU= 14.875
23:24:43 STEP 2428 H=2428:00 +CPU= 10.743
23:24:53 STEP 2429 H=2429:00 +CPU= 10.014
23:25:09 STEP 2430 H=2430:00 +CPU= 16.518
23:25:20 STEP 2431 H=2431:00 +CPU= 10.104
23:25:30 STEP 2432 H=2432:00 +CPU= 10.377
23:25:46 STEP 2433 H=2433:00 +CPU= 15.970
23:25:56 STEP 2434 H=2434:00 +CPU= 10.038
23:26:06 STEP 2435 H=2435:00 +CPU= 9.494
23:26:22 STEP 2436 H=2436:00 +CPU= 16.452
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002436
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002436
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002436
23:26:32 STEP 2437 H=2437:00 +CPU= 9.386
23:26:41 STEP 2438 H=2438:00 +CPU= 9.437
23:26:56 STEP 2439 H=2439:00 +CPU= 14.640
23:27:05 STEP 2440 H=2440:00 +CPU= 9.333
23:27:15 STEP 2441 H=2441:00 +CPU= 9.805
23:27:32 STEP 2442 H=2442:00 +CPU= 16.182
23:27:44 STEP 2443 H=2443:00 +CPU= 11.911
23:27:56 STEP 2444 H=2444:00 +CPU= 12.102
23:28:10 STEP 2445 H=2445:00 +CPU= 14.605
23:28:20 STEP 2446 H=2446:00 +CPU= 9.210
23:28:29 STEP 2447 H=2447:00 +CPU= 9.451
23:28:47 STEP 2448 H=2448:00 +CPU= 17.324
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002448
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002448
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002448
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002424
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002424
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002424
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002436
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002436
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002436
Zipping up the intermediate file: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0235_2018050100_123_987_12203879_0_r770809535_5.zip
Uploading the intermediate file: upload_file_5.zip
23:28:56 STEP 2449 H=2449:00 +CPU= 9.514
23:29:06 STEP 2450 H=2450:00 +CPU= 9.982
23:29:21 STEP 2451 H=2451:00 +CPU= 14.969
23:29:31 STEP 2452 H=2452:00 +CPU= 9.751
23:29:41 STEP 2453 H=2453:00 +CPU= 10.023
23:29:57 STEP 2454 H=2454:00 +CPU= 15.587
23:30:07 STEP 2455 H=2455:00 +CPU= 9.935
23:30:17 STEP 2456 H=2456:00 +CPU= 10.419
23:30:33 STEP 2457 H=2457:00 +CPU= 15.869
23:30:43 STEP 2458 H=2458:00 +CPU= 10.110
23:30:53 STEP 2459 H=2459:00 +CPU= 9.855
23:31:11 STEP 2460 H=2460:00 +CPU= 17.332
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002460
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002460
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002460
23:31:21 STEP 2461 H=2461:00 +CPU= 9.779
23:31:31 STEP 2462 H=2462:00 +CPU= 10.103
23:31:47 STEP 2463 H=2463:00 +CPU= 15.580
23:31:57 STEP 2464 H=2464:00 +CPU= 10.209
23:32:07 STEP 2465 H=2465:00 +CPU= 9.768
23:32:23 STEP 2466 H=2466:00 +CPU= 15.972
23:32:33 STEP 2467 H=2467:00 +CPU= 9.802
23:32:43 STEP 2468 H=2468:00 +CPU= 10.160
23:32:59 STEP 2469 H=2469:00 +CPU= 15.580
23:33:09 STEP 2470 H=2470:00 +CPU= 10.086
23:33:19 STEP 2471 H=2471:00 +CPU= 10.086
23:33:37 STEP 2472 H=2472:00 +CPU= 18.247
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002472
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002472
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002472
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002448
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002448
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002448
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002460
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002460
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002460
Zipping up the intermediate file: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0235_2018050100_123_987_12203879_0_r770809535_6.zip
Uploading the intermediate file: upload_file_6.zip
23:33:47 STEP 2473 H=2473:00 +CPU= 9.782
23:33:57 STEP 2474 H=2474:00 +CPU= 10.041
23:34:13 STEP 2475 H=2475:00 +CPU= 15.605
23:34:23 STEP 2476 H=2476:00 +CPU= 10.102
23:34:33 STEP 2477 H=2477:00 +CPU= 9.914
23:34:49 STEP 2478 H=2478:00 +CPU= 15.997
23:34:59 STEP 2479 H=2479:00 +CPU= 9.770
23:35:09 STEP 2480 H=2480:00 +CPU= 10.085
23:35:26 STEP 2481 H=2481:00 +CPU= 16.377
23:35:36 STEP 2482 H=2482:00 +CPU= 10.171
23:35:46 STEP 2483 H=2483:00 +CPU= 9.778
23:36:03 STEP 2484 H=2484:00 +CPU= 17.429
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002484
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002484
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002484
23:36:13 STEP 2485 H=2485:00 +CPU= 9.786
23:36:23 STEP 2486 H=2486:00 +CPU= 10.128
23:36:39 STEP 2487 H=2487:00 +CPU= 15.478
23:36:49 STEP 2488 H=2488:00 +CPU= 10.163
23:36:59 STEP 2489 H=2489:00 +CPU= 9.803
23:37:15 STEP 2490 H=2490:00 +CPU= 15.977
23:37:25 STEP 2491 H=2491:00 +CPU= 9.750
23:37:35 STEP 2492 H=2492:00 +CPU= 10.081
23:37:51 STEP 2493 H=2493:00 +CPU= 15.582
23:38:01 STEP 2494 H=2494:00 +CPU= 10.106
23:38:11 STEP 2495 H=2495:00 +CPU= 10.056
23:38:30 STEP 2496 H=2496:00 +CPU= 18.387
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002496
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002496
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002496
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002472
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002472
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002472
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002484
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002484
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002484
Zipping up the intermediate file: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0235_2018050100_123_987_12203879_0_r770809535_7.zip
Uploading the intermediate file: upload_file_7.zip
23:38:40 STEP 2497 H=2497:00 +CPU= 9.805
23:38:50 STEP 2498 H=2498:00 +CPU= 10.108
23:39:06 STEP 2499 H=2499:00 +CPU= 15.545
23:39:16 STEP 2500 H=2500:00 +CPU= 10.071
23:39:26 STEP 2501 H=2501:00 +CPU= 10.507
23:39:44 STEP 2502 H=2502:00 +CPU= 17.613
23:39:55 STEP 2503 H=2503:00 +CPU= 11.038
23:40:06 STEP 2504 H=2504:00 +CPU= 10.521
23:40:21 STEP 2505 H=2505:00 +CPU= 15.080
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0235
(argv4) batchid: 987
(argv5) wuid: 12203879
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /d6/home/boinc/slots/5
Project directory is: /d6/home/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879
..mkdir for temp folder for results failed
Copying: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /d6/home/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /d6/home/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a37947bf475ca8ac2aedb9de3c7e1bfc to: /d6/home/boinc/slots/5/oifs_43r3_ps_0235_2018050100_123_987_12203879.zip
Unzipping the namelist zip file: /d6/home/boinc/slots/5/oifs_43r3_ps_0235_2018050100_123_987_12203879.zip
ic_ancil_file: ic_ancil_12203879
ifsdata_file: ifsdata_12203879
climate_data_file: clim_data_12203879
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5b73891d866a923fd582c819e7820bae to: /d6/home/boinc/slots/5/ic_ancil_12203879.zip
Unzipping the IC ancils zip file: /d6/home/boinc/slots/5/ic_ancil_12203879.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c9c301ab539750247d3396013ffc9bcb to: /d6/home/boinc/slots/5/ifsdata/ifsdata_12203879.zip
Unzipping the ifsdata_zip file: /d6/home/boinc/slots/5/ifsdata/ifsdata_12203879.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /d6/home/boinc/slots/5/159l_2/clim_data_12203879.zip
Unzipping the climate data zip file: /d6/home/boinc/slots/5/159l_2/clim_data_12203879.zip
Checking for progress XML file: /d6/home/boinc/slots/5/progress_file_12203879.xml
Opened progress file ok : /d6/home/boinc/slots/5/progress_file_12203879.xml
-- Model is restarting --
Adjusting last_iter, 2505, to previous model restart step.
Creating progress file: /d6/home/boinc/slots/5/progress_file_12203879.xml
last_cpu_time: 4516
upload_file_number: 8
last_iter: 2495
last_upload: 8985600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0235_2018050100_123_987_12203879_0_r770809535
The child process has been launched with process id: 13029
Executing the command: /d6/home/boinc/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:13029:13029] [20221231:111329:1672481609:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
11:19:50 STEP 2496 H=2496:00 +CPU= 33.791
11:20:54 STEP 2497 H=2497:00 +CPU= 22.191
11:21:53 STEP 2498 H=2498:00 +CPU= 21.630
Suspend request received from the BOINC client, suspending the child process
11:24:03 STEP 2499 H=2499:00 +CPU= 25.875
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
11:30:19 STEP 2500 H=2500:00 +CPU= 14.421
Resuming the child process
11:31:26 STEP 2501 H=2501:00 +CPU= 16.622
11:33:19 STEP 2502 H=2502:00 +CPU= 29.014
11:34:51 STEP 2503 H=2503:00 +CPU= 21.399
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:36:10 STEP 2504 H=2504:00 +CPU= 16.138
11:37:16 STEP 2505 H=2505:00 +CPU= 30.518
11:37:55 STEP 2506 H=2506:00 +CPU= 18.819
11:38:44 STEP 2507 H=2507:00 +CPU= 19.292
11:39:49 STEP 2508 H=2508:00 +CPU= 29.714
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002508
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002508
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002508
11:40:39 STEP 2509 H=2509:00 +CPU= 17.991
11:41:33 STEP 2510 H=2510:00 +CPU= 20.896
11:42:47 STEP 2511 H=2511:00 +CPU= 28.247
11:43:52 STEP 2512 H=2512:00 +CPU= 19.867
11:44:31 STEP 2513 H=2513:00 +CPU= 16.099
11:45:33 STEP 2514 H=2514:00 +CPU= 25.447
11:46:07 STEP 2515 H=2515:00 +CPU= 15.742
11:46:42 STEP 2516 H=2516:00 +CPU= 16.906
11:49:26 STEP 2517 H=2517:00 +CPU= 25.991
11:51:05 STEP 2518 H=2518:00 +CPU= 17.919
11:51:40 STEP 2519 H=2519:00 +CPU= 19.176
11:52:55 STEP 2520 H=2520:00 +CPU= 34.617
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002520
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002520
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002520
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002496
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002496
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002496
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002508
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002508
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002508
Zipping up the intermediate file: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0235_2018050100_123_987_12203879_0_r770809535_8.zip
Uploading the intermediate file: upload_file_8.zip
11:53:36 STEP 2521 H=2521:00 +CPU= 18.892
11:54:19 STEP 2522 H=2522:00 +CPU= 17.456
11:55:26 STEP 2523 H=2523:00 +CPU= 26.129
11:56:22 STEP 2524 H=2524:00 +CPU= 20.304
11:57:20 STEP 2525 H=2525:00 +CPU= 19.382
11:58:23 STEP 2526 H=2526:00 +CPU= 27.968
11:59:01 STEP 2527 H=2527:00 +CPU= 17.820
11:59:38 STEP 2528 H=2528:00 +CPU= 17.554
12:01:06 STEP 2529 H=2529:00 +CPU= 27.890
12:01:56 STEP 2530 H=2530:00 +CPU= 20.985
12:02:35 STEP 2531 H=2531:00 +CPU= 18.685
12:04:11 STEP 2532 H=2532:00 +CPU= 30.395
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002532
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002532
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002532
12:04:53 STEP 2533 H=2533:00 +CPU= 17.490
12:05:42 STEP 2534 H=2534:00 +CPU= 21.090
12:06:55 STEP 2535 H=2535:00 +CPU= 27.953
12:07:29 STEP 2536 H=2536:00 +CPU= 17.456
12:08:29 STEP 2537 H=2537:00 +CPU= 20.281
12:09:40 STEP 2538 H=2538:00 +CPU= 29.576
12:10:25 STEP 2539 H=2539:00 +CPU= 18.880
12:11:38 STEP 2540 H=2540:00 +CPU= 20.180
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0235
(argv4) batchid: 987
(argv5) wuid: 12203879
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /d6/home/boinc/slots/5
Project directory is: /d6/home/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879
..mkdir for temp folder for results failed
Copying: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /d6/home/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /d6/home/boinc/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a37947bf475ca8ac2aedb9de3c7e1bfc to: /d6/home/boinc/slots/5/oifs_43r3_ps_0235_2018050100_123_987_12203879.zip
Unzipping the namelist zip file: /d6/home/boinc/slots/5/oifs_43r3_ps_0235_2018050100_123_987_12203879.zip
ic_ancil_file: ic_ancil_12203879
ifsdata_file: ifsdata_12203879
climate_data_file: clim_data_12203879
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5b73891d866a923fd582c819e7820bae to: /d6/home/boinc/slots/5/ic_ancil_12203879.zip
Unzipping the IC ancils zip file: /d6/home/boinc/slots/5/ic_ancil_12203879.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c9c301ab539750247d3396013ffc9bcb to: /d6/home/boinc/slots/5/ifsdata/ifsdata_12203879.zip
Unzipping the ifsdata_zip file: /d6/home/boinc/slots/5/ifsdata/ifsdata_12203879.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /d6/home/boinc/slots/5/159l_2/clim_data_12203879.zip
Unzipping the climate data zip file: /d6/home/boinc/slots/5/159l_2/clim_data_12203879.zip
Checking for progress XML file: /d6/home/boinc/slots/5/progress_file_12203879.xml
Opened progress file ok : /d6/home/boinc/slots/5/progress_file_12203879.xml
-- Model is restarting --
Adjusting last_iter, 2541, to previous model restart step.
Creating progress file: /d6/home/boinc/slots/5/progress_file_12203879.xml
last_cpu_time: 5532
upload_file_number: 9
last_iter: 2519
last_upload: 9072000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0235_2018050100_123_987_12203879_0_r770809535
The child process has been launched with process id: 5475
Executing the command: /d6/home/boinc/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:jmp01:1:1:5475:5475] [20221231:121347:1672485227:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
12:15:37 STEP 2520 H=2520:00 +CPU= 34.294
12:16:22 STEP 2521 H=2521:00 +CPU= 17.438
12:17:14 STEP 2522 H=2522:00 +CPU= 19.785
12:18:23 STEP 2523 H=2523:00 +CPU= 30.219
12:19:08 STEP 2524 H=2524:00 +CPU= 17.397
12:19:52 STEP 2525 H=2525:00 +CPU= 19.765
12:21:27 STEP 2526 H=2526:00 +CPU= 30.269
12:22:17 STEP 2527 H=2527:00 +CPU= 18.689
12:23:02 STEP 2528 H=2528:00 +CPU= 20.120
12:28:00 STEP 2529 H=2529:00 +CPU= 28.291
12:34:00 STEP 2530 H=2530:00 +CPU= 16.193
12:35:20 STEP 2531 H=2531:00 +CPU= 16.853
12:36:24 STEP 2532 H=2532:00 +CPU= 30.503
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002532
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002532
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002532
12:37:06 STEP 2533 H=2533:00 +CPU= 19.816
12:37:49 STEP 2534 H=2534:00 +CPU= 18.860
12:38:52 STEP 2535 H=2535:00 +CPU= 24.846
12:39:35 STEP 2536 H=2536:00 +CPU= 19.423
12:40:30 STEP 2537 H=2537:00 +CPU= 19.956
12:42:09 STEP 2538 H=2538:00 +CPU= 29.468
12:42:45 STEP 2539 H=2539:00 +CPU= 16.859
Suspend request received from the BOINC client, suspending the child process
12:43:58 STEP 2540 H=2540:00 +CPU= 14.805
Resuming the child process
12:44:57 STEP 2541 H=2541:00 +CPU= 24.335
Suspend request received from the BOINC client, suspending the child process
13:00:47 STEP 2542 H=2542:00 +CPU= 18.134
Resuming the child process
13:01:17 STEP 2543 H=2543:00 +CPU= 15.039
13:02:41 STEP 2544 H=2544:00 +CPU= 29.676
Moving to projects directory: /d6/home/boinc/slots/5/ICMGGhq0f+002544
Moving to projects directory: /d6/home/boinc/slots/5/ICMSHhq0f+002544
Moving to projects directory: /d6/home/boinc/slots/5/ICMUAhq0f+002544
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002520
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002520
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002520
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMGGhq0f+002532
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMSHhq0f+002532
Adding to the zip: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203879/ICMUAhq0f+002532
Zipping up the intermediate file: /d6/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0235_2018050100_123_987_12203879_0_r770809535_9.zip
Uploading the intermediate file: upload_file_9.zip
13:03:16 STEP 2545 H=2545:00 +CPU= 16.789
13:04:00 STEP 2546 H=2546:00 +CPU= 17.355
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
