climateprediction.net home page
Task 22303014

Task 22303014

Name oifs_43r3_ps_0181_2003050100_123_972_12188825_1
Workunit 12188825
Created 24 Jan 2023, 17:04:32 UTC
Sent 24 Jan 2023, 17:35:43 UTC
Report deadline 25 Mar 2023, 17:35:43 UTC
Received 25 Jan 2023, 16:02:01 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1538127
Run time 1 hours 16 min
CPU time 1 hours 16 min
Validate state Invalid
Credit 1,814.40
Device peak FLOPS 7.48 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,408.35 MB
Peak swap size 4,978.07 MB
Peak disk usage 2,977.95 MB
Stderr
<core_client_version>7.22.0</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
he child process
  18:57:57 STEP   79 H=  79:00 +CPU=  7.598
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:58:13 STEP   80 H=  80:00 +CPU=  7.862
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:58:33 STEP   81 H=  81:00 +CPU= 12.119
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:58:46 STEP   82 H=  82:00 +CPU=  7.758
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:58:58 STEP   83 H=  83:00 +CPU=  7.579
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:59:21 STEP   84 H=  84:00 +CPU= 13.315
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:59:33 STEP   85 H=  85:00 +CPU=  7.554
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000084
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000084
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000084
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:59:49 STEP   86 H=  86:00 +CPU=  7.826
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3992MB (heap), 4344MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x3e800000000], nsigs = 1
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3992 MB (maxheap), 4344 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190004:1674583204:1419.144] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190004:1674583204:1419.144] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000FAA5EA  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  0000000000F92699  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  0000000000F88512  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000F0B683  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0181
(argv4) batchid: 972
(argv5) wuid: 12188825
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/stefan/slots/4
Project directory is: /home/stefan/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825
..mkdir for temp folder for results failed
Copying: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/stefan/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/stefan/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_943de426e6fe821666cd1e1c0f309465 to: /home/stefan/slots/4/oifs_43r3_ps_0181_2003050100_123_972_12188825.zip
Unzipping the namelist zip file: /home/stefan/slots/4/oifs_43r3_ps_0181_2003050100_123_972_12188825.zip
ic_ancil_file: ic_ancil_12188825
ifsdata_file: ifsdata_12188825
climate_data_file: clim_data_12188825
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_170d1551d97285d1db6fa8a53abf160c to: /home/stefan/slots/4/ic_ancil_12188825.zip
Unzipping the IC ancils zip file: /home/stefan/slots/4/ic_ancil_12188825.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b440f6f96d74acf6a7feb76499bffa6b to: /home/stefan/slots/4/ifsdata/ifsdata_12188825.zip
Unzipping the ifsdata_zip file: /home/stefan/slots/4/ifsdata/ifsdata_12188825.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/stefan/slots/4/159l_2/clim_data_12188825.zip
Unzipping the climate data zip file: /home/stefan/slots/4/159l_2/clim_data_12188825.zip
Checking for progress XML file: /home/stefan/slots/4/progress_file_12188825.xml
Opened progress file ok : /home/stefan/slots/4/progress_file_12188825.xml
-- Model is restarting --
Adjusting last_iter, 87, to previous model restart step.
Creating progress file: /home/stefan/slots/4/progress_file_12188825.xml
last_cpu_time: 837
upload_file_number: 3
last_iter: 71
last_upload: 259200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030
The child process has been launched with process id: 95
Executing the command: /home/stefan/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at 0x1)
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:39:01 STEP   72 H=  72:00 +CPU= 13.496
  15:39:09 STEP   73 H=  73:00 +CPU=  7.590
  15:39:16 STEP   74 H=  74:00 +CPU=  7.730
  15:39:28 STEP   75 H=  75:00 +CPU= 12.080
  15:39:36 STEP   76 H=  76:00 +CPU=  7.782
  15:39:44 STEP   77 H=  77:00 +CPU=  7.505
  15:39:56 STEP   78 H=  78:00 +CPU= 12.211
  15:40:04 STEP   79 H=  79:00 +CPU=  7.885
  15:40:12 STEP   80 H=  80:00 +CPU=  8.095
  15:40:25 STEP   81 H=  81:00 +CPU= 12.578
  15:40:33 STEP   82 H=  82:00 +CPU=  8.106
  15:40:41 STEP   83 H=  83:00 +CPU=  7.912
  15:40:55 STEP   84 H=  84:00 +CPU= 13.883
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000084
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000084
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000084
  15:41:03 STEP   85 H=  85:00 +CPU=  7.915
  15:41:11 STEP   86 H=  86:00 +CPU=  8.146
  15:41:24 STEP   87 H=  87:00 +CPU= 12.567
  15:41:32 STEP   88 H=  88:00 +CPU=  8.161
  15:41:40 STEP   89 H=  89:00 +CPU=  7.971
  15:41:53 STEP   90 H=  90:00 +CPU= 12.826
  15:42:01 STEP   91 H=  91:00 +CPU=  7.967
  15:42:09 STEP   92 H=  92:00 +CPU=  8.161
  15:42:22 STEP   93 H=  93:00 +CPU= 12.613
  15:42:30 STEP   94 H=  94:00 +CPU=  8.186
  15:42:39 STEP   95 H=  95:00 +CPU=  8.193
  15:42:54 STEP   96 H=  96:00 +CPU= 14.988
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000096
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000096
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000096
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000072
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000072
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000072
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000084
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000084
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000084
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_3.zip
Uploading the intermediate file: upload_file_3.zip
  15:43:02 STEP   97 H=  97:00 +CPU=  7.925
  15:43:10 STEP   98 H=  98:00 +CPU=  8.121
  15:43:23 STEP   99 H=  99:00 +CPU= 12.592
  15:43:31 STEP  100 H= 100:00 +CPU=  8.139
  15:43:39 STEP  101 H= 101:00 +CPU=  7.928
  15:43:52 STEP  102 H= 102:00 +CPU= 12.871
  15:44:00 STEP  103 H= 103:00 +CPU=  7.939
  15:44:08 STEP  104 H= 104:00 +CPU=  8.136
  15:44:21 STEP  105 H= 105:00 +CPU= 12.595
  15:44:29 STEP  106 H= 106:00 +CPU=  8.142
  15:44:37 STEP  107 H= 107:00 +CPU=  7.916
  15:44:51 STEP  108 H= 108:00 +CPU= 13.950
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000108
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000108
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000108
  15:44:59 STEP  109 H= 109:00 +CPU=  7.990
  15:45:08 STEP  110 H= 110:00 +CPU=  8.202
  15:45:20 STEP  111 H= 111:00 +CPU= 12.599
  15:45:29 STEP  112 H= 112:00 +CPU=  8.160
  15:45:37 STEP  113 H= 113:00 +CPU=  7.956
  15:45:50 STEP  114 H= 114:00 +CPU= 12.870
  15:45:58 STEP  115 H= 115:00 +CPU=  7.976
  15:46:06 STEP  116 H= 116:00 +CPU=  8.156
  15:46:19 STEP  117 H= 117:00 +CPU= 12.585
  15:46:27 STEP  118 H= 118:00 +CPU=  8.155
  15:46:35 STEP  119 H= 119:00 +CPU=  8.150
  15:46:50 STEP  120 H= 120:00 +CPU= 14.992
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000120
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000120
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000120
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000096
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000096
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000096
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000108
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000108
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000108
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_4.zip
Uploading the intermediate file: upload_file_4.zip
  15:46:58 STEP  121 H= 121:00 +CPU=  7.932
  15:47:06 STEP  122 H= 122:00 +CPU=  8.146
  15:47:19 STEP  123 H= 123:00 +CPU= 12.627
  15:47:27 STEP  124 H= 124:00 +CPU=  8.165
  15:47:35 STEP  125 H= 125:00 +CPU=  7.953
  15:47:48 STEP  126 H= 126:00 +CPU= 12.845
  15:47:57 STEP  127 H= 127:00 +CPU=  8.029
  15:48:05 STEP  128 H= 128:00 +CPU=  8.196
  15:48:18 STEP  129 H= 129:00 +CPU= 12.610
  15:48:26 STEP  130 H= 130:00 +CPU=  8.153
  15:48:34 STEP  131 H= 131:00 +CPU=  7.979
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0181
(argv4) batchid: 972
(argv5) wuid: 12188825
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/stefan/slots/4
Project directory is: /home/stefan/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825
..mkdir for temp folder for results failed
Copying: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/stefan/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/stefan/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_943de426e6fe821666cd1e1c0f309465 to: /home/stefan/slots/4/oifs_43r3_ps_0181_2003050100_123_972_12188825.zip
Unzipping the namelist zip file: /home/stefan/slots/4/oifs_43r3_ps_0181_2003050100_123_972_12188825.zip
ic_ancil_file: ic_ancil_12188825
ifsdata_file: ifsdata_12188825
climate_data_file: clim_data_12188825
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_170d1551d97285d1db6fa8a53abf160c to: /home/stefan/slots/4/ic_ancil_12188825.zip
Unzipping the IC ancils zip file: /home/stefan/slots/4/ic_ancil_12188825.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b440f6f96d74acf6a7feb76499bffa6b to: /home/stefan/slots/4/ifsdata/ifsdata_12188825.zip
Unzipping the ifsdata_zip file: /home/stefan/slots/4/ifsdata/ifsdata_12188825.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/stefan/slots/4/159l_2/clim_data_12188825.zip
Unzipping the climate data zip file: /home/stefan/slots/4/159l_2/clim_data_12188825.zip
Checking for progress XML file: /home/stefan/slots/4/progress_file_12188825.xml
Opened progress file ok : /home/stefan/slots/4/progress_file_12188825.xml
-- Model is restarting --
Adjusting last_iter, 132, to previous model restart step.
Creating progress file: /home/stefan/slots/4/progress_file_12188825.xml
last_cpu_time: 1422
upload_file_number: 5
last_iter: 119
last_upload: 432000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030
The child process has been launched with process id: 81
Executing the command: /home/stefan/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at 0x1)
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:50:32 STEP  120 H= 120:00 +CPU= 14.257
  15:50:40 STEP  121 H= 121:00 +CPU=  8.040
  15:50:48 STEP  122 H= 122:00 +CPU=  8.258
  15:51:01 STEP  123 H= 123:00 +CPU= 12.756
  15:51:09 STEP  124 H= 124:00 +CPU=  8.207
  15:51:18 STEP  125 H= 125:00 +CPU=  8.076
  15:51:31 STEP  126 H= 126:00 +CPU= 12.990
  15:51:39 STEP  127 H= 127:00 +CPU=  8.035
  15:51:47 STEP  128 H= 128:00 +CPU=  8.227
  15:52:00 STEP  129 H= 129:00 +CPU= 12.711
  15:52:08 STEP  130 H= 130:00 +CPU=  8.248
  15:52:16 STEP  131 H= 131:00 +CPU=  8.053
  15:52:31 STEP  132 H= 132:00 +CPU= 14.046
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000132
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000132
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000132
  15:52:39 STEP  133 H= 133:00 +CPU=  8.050
  15:52:47 STEP  134 H= 134:00 +CPU=  8.253
  15:53:00 STEP  135 H= 135:00 +CPU= 12.741
  15:53:08 STEP  136 H= 136:00 +CPU=  8.248
  15:53:17 STEP  137 H= 137:00 +CPU=  8.039
  15:53:30 STEP  138 H= 138:00 +CPU= 13.031
  15:53:38 STEP  139 H= 139:00 +CPU=  8.063
  15:53:46 STEP  140 H= 140:00 +CPU=  8.221
  15:53:59 STEP  141 H= 141:00 +CPU= 12.694
  15:54:07 STEP  142 H= 142:00 +CPU=  8.220
  15:54:16 STEP  143 H= 143:00 +CPU=  8.233
  15:54:31 STEP  144 H= 144:00 +CPU= 15.146
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000144
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000144
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000144
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000120
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000120
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000120
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000132
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000132
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000132
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_5.zip
Uploading the intermediate file: upload_file_5.zip
  15:54:39 STEP  145 H= 145:00 +CPU=  8.033
  15:54:48 STEP  146 H= 146:00 +CPU=  8.254
  15:55:01 STEP  147 H= 147:00 +CPU= 12.765
  15:55:09 STEP  148 H= 148:00 +CPU=  8.240
  15:55:17 STEP  149 H= 149:00 +CPU=  8.021
  15:55:30 STEP  150 H= 150:00 +CPU= 12.935
  15:55:38 STEP  151 H= 151:00 +CPU=  8.032
  15:55:47 STEP  152 H= 152:00 +CPU=  8.243
  15:56:00 STEP  153 H= 153:00 +CPU= 12.793
  15:56:08 STEP  154 H= 154:00 +CPU=  8.235
  15:56:16 STEP  155 H= 155:00 +CPU=  8.034
  15:56:30 STEP  156 H= 156:00 +CPU= 14.020
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000156
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000156
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000156
  15:56:38 STEP  157 H= 157:00 +CPU=  8.039
  15:56:47 STEP  158 H= 158:00 +CPU=  8.204
  15:57:00 STEP  159 H= 159:00 +CPU= 12.720
  15:57:08 STEP  160 H= 160:00 +CPU=  8.280
  15:57:16 STEP  161 H= 161:00 +CPU=  8.055
  15:57:29 STEP  162 H= 162:00 +CPU= 12.977
  15:57:37 STEP  163 H= 163:00 +CPU=  8.065
  15:57:46 STEP  164 H= 164:00 +CPU=  8.267
  15:57:59 STEP  165 H= 165:00 +CPU= 12.781
  15:58:07 STEP  166 H= 166:00 +CPU=  8.231
  15:58:15 STEP  167 H= 167:00 +CPU=  8.224
  15:58:31 STEP  168 H= 168:00 +CPU= 15.229
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000168
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000168
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000168
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000144
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000144
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000144
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000156
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000156
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000156
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_6.zip
Uploading the intermediate file: upload_file_6.zip
  15:58:39 STEP  169 H= 169:00 +CPU=  7.997
  15:58:47 STEP  170 H= 170:00 +CPU=  8.198
  15:59:00 STEP  171 H= 171:00 +CPU= 12.697
  15:59:08 STEP  172 H= 172:00 +CPU=  8.211
  15:59:16 STEP  173 H= 173:00 +CPU=  7.716
  15:59:29 STEP  174 H= 174:00 +CPU= 12.443
  15:59:36 STEP  175 H= 175:00 +CPU=  7.699
  15:59:44 STEP  176 H= 176:00 +CPU=  7.844
  15:59:56 STEP  177 H= 177:00 +CPU= 12.192
  16:00:04 STEP  178 H= 178:00 +CPU=  7.855
  16:00:12 STEP  179 H= 179:00 +CPU=  7.660
  16:00:25 STEP  180 H= 180:00 +CPU= 13.364
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000180
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000180
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000180
  16:00:33 STEP  181 H= 181:00 +CPU=  7.621
  16:00:41 STEP  182 H= 182:00 +CPU=  7.791
  16:00:53 STEP  183 H= 183:00 +CPU= 12.039
  16:01:01 STEP  184 H= 184:00 +CPU=  7.828
  16:01:08 STEP  185 H= 185:00 +CPU=  7.618
  16:01:21 STEP  186 H= 186:00 +CPU= 12.408
  16:01:28 STEP  187 H= 187:00 +CPU=  7.715
  16:01:36 STEP  188 H= 188:00 +CPU=  7.798
  16:01:48 STEP  189 H= 189:00 +CPU= 12.069
  16:01:56 STEP  190 H= 190:00 +CPU=  7.766
  16:02:04 STEP  191 H= 191:00 +CPU=  7.799
  16:02:18 STEP  192 H= 192:00 +CPU= 14.598
  16:02:26 STEP  193 H= 193:00 +CPU=  7.687
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000192
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000192
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000192
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000168
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000168
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000168
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000180
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000180
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000180
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_7.zip
Uploading the intermediate file: upload_file_7.zip
  16:02:34 STEP  194 H= 194:00 +CPU=  7.927
  16:02:46 STEP  195 H= 195:00 +CPU= 12.413
  16:02:54 STEP  196 H= 196:00 +CPU=  7.917
  16:03:02 STEP  197 H= 197:00 +CPU=  7.663
  16:03:14 STEP  198 H= 198:00 +CPU= 12.522
  16:03:22 STEP  199 H= 199:00 +CPU=  7.585
  16:03:30 STEP  200 H= 200:00 +CPU=  7.797
  16:03:42 STEP  201 H= 201:00 +CPU= 12.176
  16:03:50 STEP  202 H= 202:00 +CPU=  7.929
  16:03:58 STEP  203 H= 203:00 +CPU=  7.781
  16:04:11 STEP  204 H= 204:00 +CPU= 13.555
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000204
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000204
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000204
  16:04:19 STEP  205 H= 205:00 +CPU=  7.695
  16:04:27 STEP  206 H= 206:00 +CPU=  8.034
  16:04:40 STEP  207 H= 207:00 +CPU= 12.762
  16:04:48 STEP  208 H= 208:00 +CPU=  8.276
  16:04:57 STEP  209 H= 209:00 +CPU=  8.032
  16:05:10 STEP  210 H= 210:00 +CPU= 13.006
  16:05:18 STEP  211 H= 211:00 +CPU=  8.056
  16:05:26 STEP  212 H= 212:00 +CPU=  8.286
  16:05:39 STEP  213 H= 213:00 +CPU= 12.743
  16:05:47 STEP  214 H= 214:00 +CPU=  8.259
  16:05:56 STEP  215 H= 215:00 +CPU=  8.196
  16:06:11 STEP  216 H= 216:00 +CPU= 15.279
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000216
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000216
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000216
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000192
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000192
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000192
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000204
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000204
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000204
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_8.zip
Uploading the intermediate file: upload_file_8.zip
  16:06:19 STEP  217 H= 217:00 +CPU=  8.069
  16:06:28 STEP  218 H= 218:00 +CPU=  8.264
  16:06:41 STEP  219 H= 219:00 +CPU= 12.778
  16:06:49 STEP  220 H= 220:00 +CPU=  8.244
  16:06:57 STEP  221 H= 221:00 +CPU=  8.044
  16:07:10 STEP  222 H= 222:00 +CPU= 13.013
  16:07:19 STEP  223 H= 223:00 +CPU=  8.074
  16:07:27 STEP  224 H= 224:00 +CPU=  8.255
  16:07:40 STEP  225 H= 225:00 +CPU= 12.744
  16:07:48 STEP  226 H= 226:00 +CPU=  8.280
  16:07:56 STEP  227 H= 227:00 +CPU=  8.075
  16:08:11 STEP  228 H= 228:00 +CPU= 14.073
  16:08:19 STEP  229 H= 229:00 +CPU=  8.054
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000228
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000228
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000228
  16:08:27 STEP  230 H= 230:00 +CPU=  8.249
  16:08:40 STEP  231 H= 231:00 +CPU= 12.765
  16:08:48 STEP  232 H= 232:00 +CPU=  8.281
  16:08:56 STEP  233 H= 233:00 +CPU=  8.032
  16:09:10 STEP  234 H= 234:00 +CPU= 12.957
  16:09:18 STEP  235 H= 235:00 +CPU=  8.049
  16:09:26 STEP  236 H= 236:00 +CPU=  8.211
  16:09:39 STEP  237 H= 237:00 +CPU= 12.733
  16:09:47 STEP  238 H= 238:00 +CPU=  8.261
  16:09:55 STEP  239 H= 239:00 +CPU=  8.117
  16:10:10 STEP  240 H= 240:00 +CPU= 14.846
  16:10:18 STEP  241 H= 241:00 +CPU=  7.861
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000240
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000240
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000240
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000216
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000216
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000216
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000228
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000228
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000228
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_9.zip
Uploading the intermediate file: upload_file_9.zip
  16:10:26 STEP  242 H= 242:00 +CPU=  8.055
  16:10:39 STEP  243 H= 243:00 +CPU= 12.414
  16:10:47 STEP  244 H= 244:00 +CPU=  8.000
  16:10:55 STEP  245 H= 245:00 +CPU=  7.783
  16:11:07 STEP  246 H= 246:00 +CPU= 12.666
  16:11:15 STEP  247 H= 247:00 +CPU=  7.813
  16:11:23 STEP  248 H= 248:00 +CPU=  8.076
  16:11:36 STEP  249 H= 249:00 +CPU= 12.407
  16:11:44 STEP  250 H= 250:00 +CPU=  8.017
  16:11:52 STEP  251 H= 251:00 +CPU=  7.829
  16:12:05 STEP  252 H= 252:00 +CPU= 13.735
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000252
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000252
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000252
  16:12:13 STEP  253 H= 253:00 +CPU=  7.904
  16:12:21 STEP  254 H= 254:00 +CPU=  8.049
  16:12:34 STEP  255 H= 255:00 +CPU= 12.434
  16:12:42 STEP  256 H= 256:00 +CPU=  8.031
  16:12:50 STEP  257 H= 257:00 +CPU=  7.846
  16:13:02 STEP  258 H= 258:00 +CPU= 12.591
  16:13:10 STEP  259 H= 259:00 +CPU=  7.835
  16:13:18 STEP  260 H= 260:00 +CPU=  7.997
  16:13:31 STEP  261 H= 261:00 +CPU= 12.382
  16:13:39 STEP  262 H= 262:00 +CPU=  8.073
  16:13:47 STEP  263 H= 263:00 +CPU=  8.004
  16:14:02 STEP  264 H= 264:00 +CPU= 14.824
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000264
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000264
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000264
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000240
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000240
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000240
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000252
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000252
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000252
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_10.zip
  16:14:10 STEP  265 H= 265:00 +CPU=  7.823
Uploading the intermediate file: upload_file_10.zip
  16:14:18 STEP  266 H= 266:00 +CPU=  8.047
  16:14:30 STEP  267 H= 267:00 +CPU= 12.437
  16:14:38 STEP  268 H= 268:00 +CPU=  8.085
  16:14:46 STEP  269 H= 269:00 +CPU=  7.874
  16:14:59 STEP  270 H= 270:00 +CPU= 12.672
  16:15:07 STEP  271 H= 271:00 +CPU=  7.834
  16:15:15 STEP  272 H= 272:00 +CPU=  8.068
  16:15:27 STEP  273 H= 273:00 +CPU= 12.478
  16:15:35 STEP  274 H= 274:00 +CPU=  8.037
  16:15:43 STEP  275 H= 275:00 +CPU=  7.868
  16:15:57 STEP  276 H= 276:00 +CPU= 13.734
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000276
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000276
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000276
  16:16:05 STEP  277 H= 277:00 +CPU=  7.853
  16:16:13 STEP  278 H= 278:00 +CPU=  8.068
  16:16:26 STEP  279 H= 279:00 +CPU= 12.408
  16:16:34 STEP  280 H= 280:00 +CPU=  8.035
  16:16:41 STEP  281 H= 281:00 +CPU=  7.826
  16:16:54 STEP  282 H= 282:00 +CPU= 12.589
  16:17:02 STEP  283 H= 283:00 +CPU=  7.909
  16:17:10 STEP  284 H= 284:00 +CPU=  8.055
  16:17:23 STEP  285 H= 285:00 +CPU= 12.365
  16:17:31 STEP  286 H= 286:00 +CPU=  7.974
  16:17:39 STEP  287 H= 287:00 +CPU=  7.979
  16:17:53 STEP  288 H= 288:00 +CPU= 14.905
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000288
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000288
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000288
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000264
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000264
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000264
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000276
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000276
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000276
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_11.zip
Uploading the intermediate file: upload_file_11.zip
  16:18:01 STEP  289 H= 289:00 +CPU=  7.765
  16:18:09 STEP  290 H= 290:00 +CPU=  7.975
  16:18:22 STEP  291 H= 291:00 +CPU= 12.368
  16:18:30 STEP  292 H= 292:00 +CPU=  7.988
  16:18:37 STEP  293 H= 293:00 +CPU=  7.737
  16:18:50 STEP  294 H= 294:00 +CPU= 12.621
  16:18:58 STEP  295 H= 295:00 +CPU=  7.876
  16:19:06 STEP  296 H= 296:00 +CPU=  7.982
  16:19:19 STEP  297 H= 297:00 +CPU= 12.436
  16:19:27 STEP  298 H= 298:00 +CPU=  7.970
  16:19:34 STEP  299 H= 299:00 +CPU=  7.777
  16:19:48 STEP  300 H= 300:00 +CPU= 13.702
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000300
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000300
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000300
  16:19:56 STEP  301 H= 301:00 +CPU=  7.778
  16:20:04 STEP  302 H= 302:00 +CPU=  8.086
  16:20:16 STEP  303 H= 303:00 +CPU= 12.343
  16:20:24 STEP  304 H= 304:00 +CPU=  8.072
  16:20:32 STEP  305 H= 305:00 +CPU=  7.854
  16:20:45 STEP  306 H= 306:00 +CPU= 12.632
  16:20:53 STEP  307 H= 307:00 +CPU=  7.881
  16:21:01 STEP  308 H= 308:00 +CPU=  8.050
  16:21:13 STEP  309 H= 309:00 +CPU= 12.401
  16:21:21 STEP  310 H= 310:00 +CPU=  8.060
  16:21:30 STEP  311 H= 311:00 +CPU=  8.039
  16:21:44 STEP  312 H= 312:00 +CPU= 14.816
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000312
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000312
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000312
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000288
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000288
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000288
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000300
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000300
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000300
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_12.zip
Uploading the intermediate file: upload_file_12.zip
  16:21:52 STEP  313 H= 313:00 +CPU=  7.835
  16:22:00 STEP  314 H= 314:00 +CPU=  8.050
  16:22:13 STEP  315 H= 315:00 +CPU= 12.420
  16:22:21 STEP  316 H= 316:00 +CPU=  7.992
  16:22:29 STEP  317 H= 317:00 +CPU=  7.859
  16:22:41 STEP  318 H= 318:00 +CPU= 12.622
  16:22:49 STEP  319 H= 319:00 +CPU=  7.773
  16:22:57 STEP  320 H= 320:00 +CPU=  8.064
  16:23:10 STEP  321 H= 321:00 +CPU= 12.385
  16:23:18 STEP  322 H= 322:00 +CPU=  8.031
  16:23:26 STEP  323 H= 323:00 +CPU=  7.770
  16:23:39 STEP  324 H= 324:00 +CPU= 13.685
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000324
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000324
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000324
  16:23:47 STEP  325 H= 325:00 +CPU=  7.865
  16:23:55 STEP  326 H= 326:00 +CPU=  8.090
  16:24:08 STEP  327 H= 327:00 +CPU= 12.427
  16:24:16 STEP  328 H= 328:00 +CPU=  7.990
  16:24:24 STEP  329 H= 329:00 +CPU=  7.819
  16:24:36 STEP  330 H= 330:00 +CPU= 12.695
  16:24:44 STEP  331 H= 331:00 +CPU=  7.857
  16:24:52 STEP  332 H= 332:00 +CPU=  7.952
  16:25:05 STEP  333 H= 333:00 +CPU= 12.381
  16:25:13 STEP  334 H= 334:00 +CPU=  7.998
  16:25:21 STEP  335 H= 335:00 +CPU=  8.077
  16:25:36 STEP  336 H= 336:00 +CPU= 14.835
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000336
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000336
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000336
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000312
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000312
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000312
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000324
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000324
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000324
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_13.zip
Uploading the intermediate file: upload_file_13.zip
  16:25:43 STEP  337 H= 337:00 +CPU=  7.848
  16:25:52 STEP  338 H= 338:00 +CPU=  8.031
  16:26:04 STEP  339 H= 339:00 +CPU= 12.432
  16:26:12 STEP  340 H= 340:00 +CPU=  8.003
  16:26:20 STEP  341 H= 341:00 +CPU=  7.841
  16:26:33 STEP  342 H= 342:00 +CPU= 12.682
  16:26:40 STEP  343 H= 343:00 +CPU=  7.829
  16:26:49 STEP  344 H= 344:00 +CPU=  8.048
  16:27:01 STEP  345 H= 345:00 +CPU= 12.456
  16:27:09 STEP  346 H= 346:00 +CPU=  8.032
  16:27:17 STEP  347 H= 347:00 +CPU=  7.814
  16:27:31 STEP  348 H= 348:00 +CPU= 13.776
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000348
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000348
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000348
  16:27:39 STEP  349 H= 349:00 +CPU=  7.846
  16:27:47 STEP  350 H= 350:00 +CPU=  7.961
  16:27:59 STEP  351 H= 351:00 +CPU= 12.433
  16:28:07 STEP  352 H= 352:00 +CPU=  8.000
  16:28:15 STEP  353 H= 353:00 +CPU=  7.842
  16:28:28 STEP  354 H= 354:00 +CPU= 12.604
  16:28:35 STEP  355 H= 355:00 +CPU=  7.801
  16:28:44 STEP  356 H= 356:00 +CPU=  8.010
  16:28:56 STEP  357 H= 357:00 +CPU= 12.372
  16:29:04 STEP  358 H= 358:00 +CPU=  8.035
  16:29:12 STEP  359 H= 359:00 +CPU=  7.985
  16:29:27 STEP  360 H= 360:00 +CPU= 14.682
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000360
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000360
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000360
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000336
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000336
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000336
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000348
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000348
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000348
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_14.zip
Uploading the intermediate file: upload_file_14.zip
  16:29:35 STEP  361 H= 361:00 +CPU=  7.786
  16:29:43 STEP  362 H= 362:00 +CPU=  8.005
Uploading trickle at timestep: 1296000
  16:29:55 STEP  363 H= 363:00 +CPU= 12.437
  16:30:03 STEP  364 H= 364:00 +CPU=  7.988
  16:30:11 STEP  365 H= 365:00 +CPU=  7.773
  16:30:24 STEP  366 H= 366:00 +CPU= 12.631
  16:30:31 STEP  367 H= 367:00 +CPU=  7.779
  16:30:39 STEP  368 H= 368:00 +CPU=  8.051
  16:30:52 STEP  369 H= 369:00 +CPU= 12.378
  16:31:00 STEP  370 H= 370:00 +CPU=  7.965
  16:31:08 STEP  371 H= 371:00 +CPU=  7.860
  16:31:21 STEP  372 H= 372:00 +CPU= 13.670
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000372
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000372
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000372
  16:31:29 STEP  373 H= 373:00 +CPU=  7.821
  16:31:37 STEP  374 H= 374:00 +CPU=  7.978
  16:31:50 STEP  375 H= 375:00 +CPU= 12.351
  16:31:58 STEP  376 H= 376:00 +CPU=  8.005
  16:32:06 STEP  377 H= 377:00 +CPU=  7.764
  16:32:18 STEP  378 H= 378:00 +CPU= 12.653
  16:32:26 STEP  379 H= 379:00 +CPU=  7.817
  16:32:34 STEP  380 H= 380:00 +CPU=  8.029
  16:32:47 STEP  381 H= 381:00 +CPU= 12.360
  16:32:55 STEP  382 H= 382:00 +CPU=  8.020
  16:33:03 STEP  383 H= 383:00 +CPU=  8.037
  16:33:17 STEP  384 H= 384:00 +CPU= 14.706
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000384
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000384
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000384
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000360
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000360
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000360
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000372
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000372
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000372
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_15.zip
Uploading the intermediate file: upload_file_15.zip
  16:33:25 STEP  385 H= 385:00 +CPU=  7.809
  16:33:33 STEP  386 H= 386:00 +CPU=  8.024
  16:33:46 STEP  387 H= 387:00 +CPU= 12.365
  16:33:54 STEP  388 H= 388:00 +CPU=  7.990
  16:34:02 STEP  389 H= 389:00 +CPU=  7.820
  16:34:14 STEP  390 H= 390:00 +CPU= 12.458
  16:34:22 STEP  391 H= 391:00 +CPU=  7.795
  16:34:30 STEP  392 H= 392:00 +CPU=  8.000
  16:34:42 STEP  393 H= 393:00 +CPU= 12.380
  16:34:50 STEP  394 H= 394:00 +CPU=  8.048
  16:34:58 STEP  395 H= 395:00 +CPU=  7.882
  16:35:12 STEP  396 H= 396:00 +CPU= 13.722
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000396
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000396
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000396
  16:35:20 STEP  397 H= 397:00 +CPU=  7.830
  16:35:28 STEP  398 H= 398:00 +CPU=  8.077
  16:35:40 STEP  399 H= 399:00 +CPU= 12.479
  16:35:49 STEP  400 H= 400:00 +CPU=  8.060
  16:35:56 STEP  401 H= 401:00 +CPU=  7.836
  16:36:09 STEP  402 H= 402:00 +CPU= 12.618
  16:36:17 STEP  403 H= 403:00 +CPU=  7.856
  16:36:25 STEP  404 H= 404:00 +CPU=  8.086
  16:36:37 STEP  405 H= 405:00 +CPU= 12.291
  16:36:45 STEP  406 H= 406:00 +CPU=  8.016
  16:36:53 STEP  407 H= 407:00 +CPU=  8.012
  16:37:08 STEP  408 H= 408:00 +CPU= 14.797
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000408
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000408
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000408
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000384
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000384
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000384
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000396
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000396
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000396
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_16.zip
Uploading the intermediate file: upload_file_16.zip
  16:37:16 STEP  409 H= 409:00 +CPU=  7.789
  16:37:24 STEP  410 H= 410:00 +CPU=  8.027
  16:37:37 STEP  411 H= 411:00 +CPU= 12.446
  16:37:45 STEP  412 H= 412:00 +CPU=  8.023
  16:37:53 STEP  413 H= 413:00 +CPU=  7.786
  16:38:05 STEP  414 H= 414:00 +CPU= 12.699
  16:38:13 STEP  415 H= 415:00 +CPU=  7.805
  16:38:21 STEP  416 H= 416:00 +CPU=  8.003
  16:38:34 STEP  417 H= 417:00 +CPU= 12.415
  16:38:42 STEP  418 H= 418:00 +CPU=  8.031
  16:38:50 STEP  419 H= 419:00 +CPU=  7.788
  16:39:03 STEP  420 H= 420:00 +CPU= 13.647
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000420
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000420
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000420
  16:39:11 STEP  421 H= 421:00 +CPU=  7.818
  16:39:19 STEP  422 H= 422:00 +CPU=  8.003
  16:39:32 STEP  423 H= 423:00 +CPU= 12.379
  16:39:40 STEP  424 H= 424:00 +CPU=  7.987
  16:39:47 STEP  425 H= 425:00 +CPU=  7.834
  16:40:00 STEP  426 H= 426:00 +CPU= 12.622
  16:40:08 STEP  427 H= 427:00 +CPU=  7.745
  16:40:16 STEP  428 H= 428:00 +CPU=  7.986
  16:40:28 STEP  429 H= 429:00 +CPU= 12.404
  16:40:36 STEP  430 H= 430:00 +CPU=  7.992
  16:40:44 STEP  431 H= 431:00 +CPU=  7.992
  16:40:59 STEP  432 H= 432:00 +CPU= 14.773
Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000432
Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000432
Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000432
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000408
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000408
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000408
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000420
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000420
Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000420
Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_17.zip
Uploading the intermediate file: upload_file_17.zip
  16:41:07 STEP  433 H= 433:00 +CPU=  7.745
  16:41:15 STEP  434 H= 434:00 +CPU=  7.950
  16:41:27 STEP  435 H= 435:00 +CPU= 12.324
  16:41:35 STEP  436 H= 436:00 +CPU=  8.039
  16:41:43 STEP  437 H= 437:00 +CPU=  7.833
  16:41:56 STEP  438 H= 438:00 +CPU= 12.541
  16:42:04 STEP  439 H= 439:00 +CPU=  7.794
  16:42:12 STEP  440 H= 440:00 +CPU=  7.985
  16:42:24 STEP  441 H= 441:00 +CPU= 12.406
  16:42:32 STEP  442 H= 442:00 +CPU=  8.022
  16:42:40 STEP  443 H= 443:00 +CPU=  7.812
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:40:56 1538127 22303014 oifs_43r3_ps_0181_2003050100_123_972_12188825_1 1,296,000 3,762 0.0029


©2024 cpdn.org