climateprediction.net home page
Task 22315450

Task 22315450

Name oifs_43r3_00i7_2019110100_123_993_12214093_0
Workunit 12214093
Created 24 Feb 2023, 13:47:22 UTC
Sent 25 Feb 2023, 0:47:52 UTC
Report deadline 26 Apr 2023, 0:47:52 UTC
Received 25 Feb 2023, 19:58:42 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1532237
Run time 16 hours 43 min 42 sec
CPU time 16 hours 43 min 42 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 7.50 GFLOPS
Application version OpenIFS 43r3 v1.21
x86_64-pc-linux-gnu
Peak working set size 4,421.61 MB
Peak swap size 4,983.87 MB
Peak disk usage 1,264.60 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
/slots/13/ICMSHhnw4+002652
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002652
  09:43:19 STEP 2653 H=2653:00 +CPU= 17.082
  09:43:37 STEP 2654 H=2654:00 +CPU= 18.150
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  7.18728490733962E-03 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: evolving YSPSDT_AR1(  1)
  MAXGPFV : MAX. VALUE =    1.14121313050017     
  MAXGPFV : MAX. VALUE =   0.742457216192849     
 NSTEP =  2652 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGhnw4+002652 MODE=a
 IO-STREAM CLOSED - ICMGGhnw4+002652
 MPI-TASK:   1 -    317787578 BYTES IN      8 RECORDS TRANSFERRED IN    0.0005 SECONDS635575.2 Mbytes/s, TOTAL TIME=    0.0102( 4.9%)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219111500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219113000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219114500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219120000  !!!!!!!!!!!! 
 09:43:02 STEP 2652 H=2652:00 +CPU= 27.434
 NSTEP =  2653 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.318416112970E-01 1.765723768639E-01 6.278363225641E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.556139469209E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -1.09708217146346E-02 -2.13106835828315E+00  1.70535669844666E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -7.05393322482655E-03 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: evolving YSPSDT_AR1(  1)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219121500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219123000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219124500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219130000  !!!!!!!!!!!! 
 09:43:19 STEP 2653 H=2653:00 +CPU= 17.082
 NSTEP =  2654 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.286022384548E-01 1.699888437689E-01 6.305407492103E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.557304518810E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -1.00347928723936E-03 -1.76944258769318E+00  1.71359301934096E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  1.72979502524533E-03 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: evolving YSPSDT_AR1(  1)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219131500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219133000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219134500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200219140000  !!!!!!!!!!!! 
 09:43:37 STEP 2654 H=2654:00 +CPU= 18.150
ISTEP=    2655 RSOLINC= 1360.9440 RI0= 1366.0000 ZI0= 1360.9440 RIP0= 1392.8295
ISTEP=    2655 ZI0= 1360.9440 ZSEASON= 1.023428962 REA=149597870000.00 RDEASO=147875613210.48
  DATE= 2020    2   19
ISTEP=    2655: Reset RSOLINC= 1360.9440 RIP0= 1392.8295
 SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2:   6.248620232976000E-004
  9.609089255168670E-007  4.998719107099000E-007  9.565112577734999E-010
  2.022325970022666E-009  7.940283000000001E-011
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20191101
      159390  1.159977703455965E-004
SUECSO4: USE AVAILABLE FILES FOR DECADES:   2010 2020
 SUBROUTINE SUECSO4:, UNWEIGHTED AVERAGE:   2.346239159279681E-003
 NSTEP =  2655 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.133510651720E-01 1.702471583074E-01 6.409033008344E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.371305476021E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  2.42861266851737E-03 -1.79108619275667E+00  2.05778009301804E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  1.97520950858256E-03 -1.00000000000000E+00  1.00000000000000E+00
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 09:42:35 A00000000 STEPO     2652    16.237   16.237   16.330    149:46    150:56 0.11916182795696E-04       2GB       0MB
 09:42:35 0AA000000 STEPO     2652     0.049    0.049    0.049    149:46    150:56 0.11916182795696E-04       2GB       0MB
 09:42:35 FULLPOS-B DYNFPOS   2652     0.679    0.679    0.684    149:47    150:56 0.11916182795696E-04       2GB       0MB
 09:42:36 0AAA00AAA STEPO     2652     0.832    0.832    0.837    149:48    150:57 0.11916182795696E-04       2GB       0MB
 09:43:02 0AAA00AAA STEPO     2653    25.735   25.735   25.888    150:13    151:23 0.11916182795696E-04       2GB       0MB
 09:43:19 0AAA00AAA STEPO     2654    17.082   17.082   17.176    150:30    151:40 0.11916182795696E-04       2GB       0MB
 09:43:37 0AAA00AAA STEPO     2655    18.151   18.151   18.238    150:49    151:59 0.11916182795696E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    2640",
 CTIME   = "01100000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 5*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =   98321.2920938529     ,
 GMASSI  =   98320.7692244569     
 /
------------------------------------------------
>>> Printing last 17 lines from file: waminfo

PARAM=10U/10V,                                                          
REPRES=GG,                                                              
ANALYSIS PERIOD =       0                                               
FORECAST PERIOD =    9072000                                            
WIND TIME STEP IN SECONDS =    3600                                     
ANALYSIS FROM 20191101000000 TO 20191101000000                          
ASSIMILATION NO                                                         
NEW FORECAST NO                                                         
STATUS STORM1= UNFINISHED                                               
STATUS STORM2= UNFINISHED                                               
STATUS STORM3= UNFINISHED                                               
DATE FOR OUTPUT OF BOTH RESTART FILES = 00000000000000                  
LAST DATE FOR SPECTRA FILE OUTPUT = 20200303000000                      
BEGIN DATE FOR USING SURFACE CURRENT = 20191101000000                   
PARALLEL RESTART = YES                                                  
NUMBER OF MPI TASKS USED =          1                                   
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/13/progress_file_12214093.xml

<running_values>
  <last_cpu_time>53858.130000</last_cpu_time>
  <upload_file_number>110</upload_file_number>
  <last_iter>2655</last_iter>
  <last_upload>9504000</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 00i7
(argv4) batchid: 993
(argv5) wuid: 12214093
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/13
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/13/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/13/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9a9b337986bd20f8ff685095f382b014 to: /var/lib/boinc-client/slots/13/oifs_43r3_00i7_2019110100_123_993_12214093.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/13/oifs_43r3_00i7_2019110100_123_993_12214093.zip
ic_ancil_file: ic_ancil_12214093
ifsdata_file: ifsdata_12214093
climate_data_file: clim_data_12214093
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_32113beb410d7a931374ad91edc625b6 to: /var/lib/boinc-client/slots/13/ic_ancil_12214093.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/13/ic_ancil_12214093.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_81511501ca075a7ed181bb7a8fd5f9e4 to: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12214093.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12214093.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/13/159l_2/clim_data_12214093.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/13/159l_2/clim_data_12214093.zip
Checking for progress XML file: /var/lib/boinc-client/slots/13/progress_file_12214093.xml
Opened progress file ok : /var/lib/boinc-client/slots/13/progress_file_12214093.xml
-- Model is restarting --
Adjusting last_iter, 2655, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/13/progress_file_12214093.xml
last_cpu_time: 53858
upload_file_number: 110
last_iter: 2641
last_upload: 9504000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722
The child process has been launched with process id: 570965
Executing the command: /var/lib/boinc-client/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:570965:570965] [20230225:094523:1677343523:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  09:45:58 STEP 2640 H=2640:00 +CPU= 31.878
  09:46:15 STEP 2641 H=2641:00 +CPU= 17.042
  09:46:31 STEP 2642 H=2642:00 +CPU= 16.484
  09:46:58 STEP 2643 H=2643:00 +CPU= 26.528
  09:47:16 STEP 2644 H=2644:00 +CPU= 17.807
  09:47:32 STEP 2645 H=2645:00 +CPU= 16.012
  09:47:59 STEP 2646 H=2646:00 +CPU= 26.489
  09:48:16 STEP 2647 H=2647:00 +CPU= 16.562
  09:48:33 STEP 2648 H=2648:00 +CPU= 16.832
  09:48:59 STEP 2649 H=2649:00 +CPU= 26.226
  09:49:17 STEP 2650 H=2650:00 +CPU= 17.447
  09:49:33 STEP 2651 H=2651:00 +CPU= 15.878
  09:50:01 STEP 2652 H=2652:00 +CPU= 27.920
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002652
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002652
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002652
  09:50:17 STEP 2653 H=2653:00 +CPU= 16.366
  09:50:34 STEP 2654 H=2654:00 +CPU= 16.802
  09:51:00 STEP 2655 H=2655:00 +CPU= 25.611
  09:51:17 STEP 2656 H=2656:00 +CPU= 16.842
  09:51:33 STEP 2657 H=2657:00 +CPU= 16.135
  09:52:00 STEP 2658 H=2658:00 +CPU= 25.853
  09:52:16 STEP 2659 H=2659:00 +CPU= 16.769
  09:52:33 STEP 2660 H=2660:00 +CPU= 16.951
  09:52:59 STEP 2661 H=2661:00 +CPU= 25.549
  09:53:16 STEP 2662 H=2662:00 +CPU= 16.778
  09:53:33 STEP 2663 H=2663:00 +CPU= 16.704
  09:54:03 STEP 2664 H=2664:00 +CPU= 30.259
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002664
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002664
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_110.zip
Uploading the intermediate file: upload_file_110.zip
  09:54:20 STEP 2665 H=2665:00 +CPU= 16.729
  09:54:37 STEP 2666 H=2666:00 +CPU= 16.403
  09:55:03 STEP 2667 H=2667:00 +CPU= 26.178
  09:55:20 STEP 2668 H=2668:00 +CPU= 16.836
  09:55:36 STEP 2669 H=2669:00 +CPU= 15.713
  09:56:03 STEP 2670 H=2670:00 +CPU= 26.811
  09:56:19 STEP 2671 H=2671:00 +CPU= 16.367
  09:56:36 STEP 2672 H=2672:00 +CPU= 16.935
  09:57:02 STEP 2673 H=2673:00 +CPU= 25.586
  09:57:19 STEP 2674 H=2674:00 +CPU= 17.012
  09:57:36 STEP 2675 H=2675:00 +CPU= 16.667
  09:58:05 STEP 2676 H=2676:00 +CPU= 28.861
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002676
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002676
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002676
  09:58:22 STEP 2677 H=2677:00 +CPU= 16.949
  09:58:39 STEP 2678 H=2678:00 +CPU= 16.422
  09:59:05 STEP 2679 H=2679:00 +CPU= 25.988
  09:59:22 STEP 2680 H=2680:00 +CPU= 16.916
  09:59:39 STEP 2681 H=2681:00 +CPU= 16.623
  10:00:06 STEP 2682 H=2682:00 +CPU= 26.772
  10:00:23 STEP 2683 H=2683:00 +CPU= 17.391
  10:00:40 STEP 2684 H=2684:00 +CPU= 16.864
  10:01:06 STEP 2685 H=2685:00 +CPU= 26.058
  10:01:23 STEP 2686 H=2686:00 +CPU= 16.928
  10:01:40 STEP 2687 H=2687:00 +CPU= 16.543
  10:02:11 STEP 2688 H=2688:00 +CPU= 30.782
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002688
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002688
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_111.zip
Uploading the intermediate file: upload_file_111.zip
  10:02:28 STEP 2689 H=2689:00 +CPU= 16.730
  10:02:44 STEP 2690 H=2690:00 +CPU= 16.084
  10:03:10 STEP 2691 H=2691:00 +CPU= 25.717
  10:03:27 STEP 2692 H=2692:00 +CPU= 16.697
  10:03:43 STEP 2693 H=2693:00 +CPU= 16.106
  10:04:11 STEP 2694 H=2694:00 +CPU= 27.690
  10:04:29 STEP 2695 H=2695:00 +CPU= 17.988
  10:04:47 STEP 2696 H=2696:00 +CPU= 17.595
  10:05:13 STEP 2697 H=2697:00 +CPU= 25.207
  10:05:29 STEP 2698 H=2698:00 +CPU= 16.533
  10:05:46 STEP 2699 H=2699:00 +CPU= 16.303
  10:06:16 STEP 2700 H=2700:00 +CPU= 29.483
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002700
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002700
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002700
  10:06:33 STEP 2701 H=2701:00 +CPU= 16.975
  10:06:50 STEP 2702 H=2702:00 +CPU= 17.055
  10:07:17 STEP 2703 H=2703:00 +CPU= 26.506
  10:07:34 STEP 2704 H=2704:00 +CPU= 17.285
  10:07:51 STEP 2705 H=2705:00 +CPU= 16.607
  10:08:17 STEP 2706 H=2706:00 +CPU= 26.292
  10:08:34 STEP 2707 H=2707:00 +CPU= 16.146
  10:08:51 STEP 2708 H=2708:00 +CPU= 17.489
  10:09:17 STEP 2709 H=2709:00 +CPU= 25.763
  10:09:34 STEP 2710 H=2710:00 +CPU= 17.115
  10:09:51 STEP 2711 H=2711:00 +CPU= 16.872
  10:10:21 STEP 2712 H=2712:00 +CPU= 29.939
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002712
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002712
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_112.zip
Uploading the intermediate file: upload_file_112.zip
  10:10:39 STEP 2713 H=2713:00 +CPU= 17.288
  10:10:56 STEP 2714 H=2714:00 +CPU= 16.589
  10:11:21 STEP 2715 H=2715:00 +CPU= 25.381
  10:11:38 STEP 2716 H=2716:00 +CPU= 16.880
  10:11:54 STEP 2717 H=2717:00 +CPU= 16.257
  10:12:21 STEP 2718 H=2718:00 +CPU= 26.684
  10:12:38 STEP 2719 H=2719:00 +CPU= 16.513
  10:12:54 STEP 2720 H=2720:00 +CPU= 16.389
  10:13:20 STEP 2721 H=2721:00 +CPU= 25.656
  10:13:38 STEP 2722 H=2722:00 +CPU= 17.782
  10:13:55 STEP 2723 H=2723:00 +CPU= 16.338
  10:14:24 STEP 2724 H=2724:00 +CPU= 28.749
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002724
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002724
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002724
  10:14:40 STEP 2725 H=2725:00 +CPU= 16.467
  10:14:58 STEP 2726 H=2726:00 +CPU= 17.198
  10:15:25 STEP 2727 H=2727:00 +CPU= 26.514
  10:15:42 STEP 2728 H=2728:00 +CPU= 17.225
  10:15:59 STEP 2729 H=2729:00 +CPU= 16.849
  10:16:25 STEP 2730 H=2730:00 +CPU= 26.199
  10:16:42 STEP 2731 H=2731:00 +CPU= 16.931
  10:16:59 STEP 2732 H=2732:00 +CPU= 16.426
  10:17:25 STEP 2733 H=2733:00 +CPU= 26.055
  10:17:42 STEP 2734 H=2734:00 +CPU= 17.248
  10:17:59 STEP 2735 H=2735:00 +CPU= 16.770
  10:18:30 STEP 2736 H=2736:00 +CPU= 30.249
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002736
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002736
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_113.zip
Uploading the intermediate file: upload_file_113.zip
  10:18:46 STEP 2737 H=2737:00 +CPU= 16.091
  10:19:03 STEP 2738 H=2738:00 +CPU= 16.969
  10:19:29 STEP 2739 H=2739:00 +CPU= 25.931
  10:19:46 STEP 2740 H=2740:00 +CPU= 17.126
  10:20:03 STEP 2741 H=2741:00 +CPU= 16.869
  10:20:30 STEP 2742 H=2742:00 +CPU= 26.393
  10:20:47 STEP 2743 H=2743:00 +CPU= 16.945
  10:21:04 STEP 2744 H=2744:00 +CPU= 16.523
  10:21:31 STEP 2745 H=2745:00 +CPU= 26.944
  10:21:48 STEP 2746 H=2746:00 +CPU= 17.054
  10:22:05 STEP 2747 H=2747:00 +CPU= 16.459
  10:22:33 STEP 2748 H=2748:00 +CPU= 28.193
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002748
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002748
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002748
  10:22:50 STEP 2749 H=2749:00 +CPU= 16.017
  10:23:06 STEP 2750 H=2750:00 +CPU= 16.847
  10:23:33 STEP 2751 H=2751:00 +CPU= 26.381
  10:23:50 STEP 2752 H=2752:00 +CPU= 16.596
  10:24:06 STEP 2753 H=2753:00 +CPU= 16.741
  10:24:33 STEP 2754 H=2754:00 +CPU= 26.202
  10:24:50 STEP 2755 H=2755:00 +CPU= 16.570
  10:25:07 STEP 2756 H=2756:00 +CPU= 17.273
  10:25:33 STEP 2757 H=2757:00 +CPU= 26.076
  10:25:50 STEP 2758 H=2758:00 +CPU= 16.340
  10:26:08 STEP 2759 H=2759:00 +CPU= 17.910
  10:26:38 STEP 2760 H=2760:00 +CPU= 29.738
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002760
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002760
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_114.zip
Uploading the intermediate file: upload_file_114.zip
  10:26:54 STEP 2761 H=2761:00 +CPU= 16.356
  10:27:12 STEP 2762 H=2762:00 +CPU= 17.916
  10:27:38 STEP 2763 H=2763:00 +CPU= 25.379
  10:27:55 STEP 2764 H=2764:00 +CPU= 17.022
  10:28:11 STEP 2765 H=2765:00 +CPU= 16.509
  10:28:38 STEP 2766 H=2766:00 +CPU= 26.467
  10:28:55 STEP 2767 H=2767:00 +CPU= 17.261
  10:29:13 STEP 2768 H=2768:00 +CPU= 16.894
  10:29:38 STEP 2769 H=2769:00 +CPU= 25.547
  10:29:55 STEP 2770 H=2770:00 +CPU= 16.919
  10:30:13 STEP 2771 H=2771:00 +CPU= 17.115
  10:30:41 STEP 2772 H=2772:00 +CPU= 27.738
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002772
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002772
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002772
  10:30:57 STEP 2773 H=2773:00 +CPU= 16.454
  10:31:14 STEP 2774 H=2774:00 +CPU= 17.016
  10:31:40 STEP 2775 H=2775:00 +CPU= 25.539
  10:31:57 STEP 2776 H=2776:00 +CPU= 17.024
  10:32:14 STEP 2777 H=2777:00 +CPU= 16.429
  10:32:40 STEP 2778 H=2778:00 +CPU= 25.495
  10:32:56 STEP 2779 H=2779:00 +CPU= 16.702
  10:33:13 STEP 2780 H=2780:00 +CPU= 16.919
  10:33:39 STEP 2781 H=2781:00 +CPU= 25.695
  10:33:56 STEP 2782 H=2782:00 +CPU= 16.734
  10:34:14 STEP 2783 H=2783:00 +CPU= 17.805
  10:34:44 STEP 2784 H=2784:00 +CPU= 30.345
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002784
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002784
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_115.zip
Uploading the intermediate file: upload_file_115.zip
  10:35:02 STEP 2785 H=2785:00 +CPU= 17.397
  10:35:19 STEP 2786 H=2786:00 +CPU= 17.062
  10:35:45 STEP 2787 H=2787:00 +CPU= 25.574
  10:36:02 STEP 2788 H=2788:00 +CPU= 17.219
  10:36:19 STEP 2789 H=2789:00 +CPU= 17.184
  10:36:45 STEP 2790 H=2790:00 +CPU= 25.682
  10:37:02 STEP 2791 H=2791:00 +CPU= 16.910
  10:37:20 STEP 2792 H=2792:00 +CPU= 17.306
  10:37:46 STEP 2793 H=2793:00 +CPU= 25.637
  10:38:03 STEP 2794 H=2794:00 +CPU= 17.076
  10:38:20 STEP 2795 H=2795:00 +CPU= 16.750
  10:38:48 STEP 2796 H=2796:00 +CPU= 27.721
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002796
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002796
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002796
  10:39:05 STEP 2797 H=2797:00 +CPU= 17.021
  10:39:23 STEP 2798 H=2798:00 +CPU= 17.374
  10:39:48 STEP 2799 H=2799:00 +CPU= 25.177
  10:40:05 STEP 2800 H=2800:00 +CPU= 17.163
  10:40:22 STEP 2801 H=2801:00 +CPU= 16.780
  10:40:48 STEP 2802 H=2802:00 +CPU= 26.128
  10:41:05 STEP 2803 H=2803:00 +CPU= 16.929
  10:41:24 STEP 2804 H=2804:00 +CPU= 17.905
  10:41:50 STEP 2805 H=2805:00 +CPU= 26.546
  10:42:07 STEP 2806 H=2806:00 +CPU= 17.013
  10:42:25 STEP 2807 H=2807:00 +CPU= 17.293
  10:42:55 STEP 2808 H=2808:00 +CPU= 29.626
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002808
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002808
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_116.zip
Uploading the intermediate file: upload_file_116.zip
  10:43:11 STEP 2809 H=2809:00 +CPU= 15.947
  10:43:27 STEP 2810 H=2810:00 +CPU= 16.049
  10:43:52 STEP 2811 H=2811:00 +CPU= 25.019
  10:44:09 STEP 2812 H=2812:00 +CPU= 17.066
  10:44:25 STEP 2813 H=2813:00 +CPU= 16.296
  10:44:52 STEP 2814 H=2814:00 +CPU= 26.471
  10:45:10 STEP 2815 H=2815:00 +CPU= 17.901
  10:45:28 STEP 2816 H=2816:00 +CPU= 17.461
  10:45:54 STEP 2817 H=2817:00 +CPU= 26.086
  10:46:11 STEP 2818 H=2818:00 +CPU= 17.036
  10:46:28 STEP 2819 H=2819:00 +CPU= 16.786
  10:46:58 STEP 2820 H=2820:00 +CPU= 29.624
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002820
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002820
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002820
  10:47:16 STEP 2821 H=2821:00 +CPU= 17.333
  10:47:33 STEP 2822 H=2822:00 +CPU= 16.639
  10:47:58 STEP 2823 H=2823:00 +CPU= 25.346
  10:48:15 STEP 2824 H=2824:00 +CPU= 16.640
  10:48:33 STEP 2825 H=2825:00 +CPU= 17.860
  10:48:59 STEP 2826 H=2826:00 +CPU= 25.920
  10:49:15 STEP 2827 H=2827:00 +CPU= 16.121
  10:49:32 STEP 2828 H=2828:00 +CPU= 16.482
  10:49:58 STEP 2829 H=2829:00 +CPU= 25.788
  10:50:15 STEP 2830 H=2830:00 +CPU= 16.795
  10:50:32 STEP 2831 H=2831:00 +CPU= 17.326
  10:51:02 STEP 2832 H=2832:00 +CPU= 29.139
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002832
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002832
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_117.zip
Uploading the intermediate file: upload_file_117.zip
  10:51:18 STEP 2833 H=2833:00 +CPU= 16.711
  10:51:35 STEP 2834 H=2834:00 +CPU= 16.179
  10:52:01 STEP 2835 H=2835:00 +CPU= 25.917
  10:52:19 STEP 2836 H=2836:00 +CPU= 18.142
  10:52:36 STEP 2837 H=2837:00 +CPU= 16.404
  10:53:02 STEP 2838 H=2838:00 +CPU= 26.027
  10:53:19 STEP 2839 H=2839:00 +CPU= 16.678
  10:53:35 STEP 2840 H=2840:00 +CPU= 16.724
  10:54:01 STEP 2841 H=2841:00 +CPU= 25.668
  10:54:18 STEP 2842 H=2842:00 +CPU= 16.873
  10:54:35 STEP 2843 H=2843:00 +CPU= 16.760
  10:55:04 STEP 2844 H=2844:00 +CPU= 28.453
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002844
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002844
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002844
  10:55:20 STEP 2845 H=2845:00 +CPU= 16.089
  10:55:37 STEP 2846 H=2846:00 +CPU= 16.313
  10:56:01 STEP 2847 H=2847:00 +CPU= 24.128
  10:56:16 STEP 2848 H=2848:00 +CPU= 15.061
  10:56:31 STEP 2849 H=2849:00 +CPU= 15.013
  10:56:55 STEP 2850 H=2850:00 +CPU= 23.551
  10:57:11 STEP 2851 H=2851:00 +CPU= 15.546
  10:57:27 STEP 2852 H=2852:00 +CPU= 15.810
  10:57:51 STEP 2853 H=2853:00 +CPU= 24.047
  10:58:06 STEP 2854 H=2854:00 +CPU= 15.367
  10:58:22 STEP 2855 H=2855:00 +CPU= 15.363
  10:58:50 STEP 2856 H=2856:00 +CPU= 27.539
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002856
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002856
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_118.zip
Uploading the intermediate file: upload_file_118.zip
  10:59:06 STEP 2857 H=2857:00 +CPU= 16.258
  10:59:22 STEP 2858 H=2858:00 +CPU= 16.248
  10:59:48 STEP 2859 H=2859:00 +CPU= 25.685
  11:00:05 STEP 2860 H=2860:00 +CPU= 16.977
  11:00:23 STEP 2861 H=2861:00 +CPU= 17.311
  11:00:49 STEP 2862 H=2862:00 +CPU= 26.538
  11:01:06 STEP 2863 H=2863:00 +CPU= 16.242
  11:01:23 STEP 2864 H=2864:00 +CPU= 16.832
  11:01:49 STEP 2865 H=2865:00 +CPU= 25.983
  11:02:06 STEP 2866 H=2866:00 +CPU= 16.636
  11:02:23 STEP 2867 H=2867:00 +CPU= 16.992
  11:02:51 STEP 2868 H=2868:00 +CPU= 28.447
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002868
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002868
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002868
  11:03:08 STEP 2869 H=2869:00 +CPU= 16.526
  11:03:26 STEP 2870 H=2870:00 +CPU= 17.506
  11:03:52 STEP 2871 H=2871:00 +CPU= 25.835
  11:04:10 STEP 2872 H=2872:00 +CPU= 17.699
  11:04:26 STEP 2873 H=2873:00 +CPU= 16.320
  11:04:52 STEP 2874 H=2874:00 +CPU= 25.452
  11:05:08 STEP 2875 H=2875:00 +CPU= 16.071
  11:05:24 STEP 2876 H=2876:00 +CPU= 16.438
  11:05:50 STEP 2877 H=2877:00 +CPU= 25.182
  11:06:07 STEP 2878 H=2878:00 +CPU= 16.777
  11:06:23 STEP 2879 H=2879:00 +CPU= 16.250
  11:06:52 STEP 2880 H=2880:00 +CPU= 28.957
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002880
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002880
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_119.zip
Uploading the intermediate file: upload_file_119.zip
  11:07:08 STEP 2881 H=2881:00 +CPU= 16.182
Uploading trickle at timestep: 10368000
  11:07:27 STEP 2882 H=2882:00 +CPU= 17.761
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 00i7
(argv4) batchid: 993
(argv5) wuid: 12214093
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/13
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/13/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/13/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9a9b337986bd20f8ff685095f382b014 to: /var/lib/boinc-client/slots/13/oifs_43r3_00i7_2019110100_123_993_12214093.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/13/oifs_43r3_00i7_2019110100_123_993_12214093.zip
ic_ancil_file: ic_ancil_12214093
ifsdata_file: ifsdata_12214093
climate_data_file: clim_data_12214093
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_32113beb410d7a931374ad91edc625b6 to: /var/lib/boinc-client/slots/13/ic_ancil_12214093.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/13/ic_ancil_12214093.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_81511501ca075a7ed181bb7a8fd5f9e4 to: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12214093.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12214093.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/13/159l_2/clim_data_12214093.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/13/159l_2/clim_data_12214093.zip
Checking for progress XML file: /var/lib/boinc-client/slots/13/progress_file_12214093.xml
Opened progress file ok : /var/lib/boinc-client/slots/13/progress_file_12214093.xml
-- Model is restarting --
Adjusting last_iter, 2883, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/13/progress_file_12214093.xml
last_cpu_time: 58757
upload_file_number: 120
last_iter: 2881
last_upload: 10368000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722
The child process has been launched with process id: 589429
Executing the command: /var/lib/boinc-client/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:589429:589429] [20230225:113952:1677350392:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:40:26 STEP 2880 H=2880:00 +CPU= 30.649
  11:40:43 STEP 2881 H=2881:00 +CPU= 16.752
  11:41:01 STEP 2882 H=2882:00 +CPU= 17.186
  11:41:28 STEP 2883 H=2883:00 +CPU= 26.915
  11:41:46 STEP 2884 H=2884:00 +CPU= 17.952
  11:42:05 STEP 2885 H=2885:00 +CPU= 17.750
  11:42:33 STEP 2886 H=2886:00 +CPU= 27.468
  11:42:51 STEP 2887 H=2887:00 +CPU= 17.334
  11:43:09 STEP 2888 H=2888:00 +CPU= 17.585
  11:43:37 STEP 2889 H=2889:00 +CPU= 27.470
  11:43:56 STEP 2890 H=2890:00 +CPU= 18.383
  11:44:14 STEP 2891 H=2891:00 +CPU= 17.697
  11:44:45 STEP 2892 H=2892:00 +CPU= 30.007
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002892
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002892
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002892
  11:45:03 STEP 2893 H=2893:00 +CPU= 17.790
  11:45:22 STEP 2894 H=2894:00 +CPU= 17.848
  11:45:50 STEP 2895 H=2895:00 +CPU= 26.962
  11:46:09 STEP 2896 H=2896:00 +CPU= 18.879
  11:46:27 STEP 2897 H=2897:00 +CPU= 17.873
  11:46:56 STEP 2898 H=2898:00 +CPU= 28.093
  11:47:14 STEP 2899 H=2899:00 +CPU= 17.670
  11:47:33 STEP 2900 H=2900:00 +CPU= 18.166
  11:48:00 STEP 2901 H=2901:00 +CPU= 26.378
  11:48:19 STEP 2902 H=2902:00 +CPU= 18.521
  11:48:38 STEP 2903 H=2903:00 +CPU= 17.936
  11:49:09 STEP 2904 H=2904:00 +CPU= 30.580
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002904
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002904
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_120.zip
Uploading the intermediate file: upload_file_120.zip
  11:49:27 STEP 2905 H=2905:00 +CPU= 17.698
  11:49:45 STEP 2906 H=2906:00 +CPU= 17.580
  11:50:12 STEP 2907 H=2907:00 +CPU= 26.596
  11:50:30 STEP 2908 H=2908:00 +CPU= 17.125
  11:50:45 STEP 2909 H=2909:00 +CPU= 15.180
  11:51:10 STEP 2910 H=2910:00 +CPU= 25.105
  11:51:26 STEP 2911 H=2911:00 +CPU= 15.728
  11:51:42 STEP 2912 H=2912:00 +CPU= 15.617
  11:52:06 STEP 2913 H=2913:00 +CPU= 24.017
  11:52:22 STEP 2914 H=2914:00 +CPU= 16.097
  11:52:37 STEP 2915 H=2915:00 +CPU= 14.791
  11:53:04 STEP 2916 H=2916:00 +CPU= 25.978
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002916
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002916
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002916
  11:53:19 STEP 2917 H=2917:00 +CPU= 15.421
  11:53:35 STEP 2918 H=2918:00 +CPU= 15.359
  11:53:58 STEP 2919 H=2919:00 +CPU= 23.462
  11:54:14 STEP 2920 H=2920:00 +CPU= 15.608
  11:54:30 STEP 2921 H=2921:00 +CPU= 15.698
  11:54:55 STEP 2922 H=2922:00 +CPU= 25.099
  11:55:12 STEP 2923 H=2923:00 +CPU= 16.383
  11:55:28 STEP 2924 H=2924:00 +CPU= 16.450
  11:55:54 STEP 2925 H=2925:00 +CPU= 24.925
  11:56:10 STEP 2926 H=2926:00 +CPU= 16.698
  11:56:27 STEP 2927 H=2927:00 +CPU= 16.395
  11:56:55 STEP 2928 H=2928:00 +CPU= 28.215
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002928
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002928
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_121.zip
Uploading the intermediate file: upload_file_121.zip
  11:57:11 STEP 2929 H=2929:00 +CPU= 15.781
  11:57:27 STEP 2930 H=2930:00 +CPU= 15.913
  11:57:52 STEP 2931 H=2931:00 +CPU= 24.517
  11:58:09 STEP 2932 H=2932:00 +CPU= 16.421
  11:58:25 STEP 2933 H=2933:00 +CPU= 15.885
  11:58:50 STEP 2934 H=2934:00 +CPU= 25.338
  11:59:06 STEP 2935 H=2935:00 +CPU= 16.206
  11:59:23 STEP 2936 H=2936:00 +CPU= 16.141
  11:59:48 STEP 2937 H=2937:00 +CPU= 24.610
  12:00:05 STEP 2938 H=2938:00 +CPU= 16.874
  12:00:21 STEP 2939 H=2939:00 +CPU= 15.893
  12:00:48 STEP 2940 H=2940:00 +CPU= 26.909
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002940
  12:01:04 STEP 2941 H=2941:00 +CPU= 15.848
  12:01:20 STEP 2942 H=2942:00 +CPU= 15.981
  12:01:45 STEP 2943 H=2943:00 +CPU= 24.396
  12:02:02 STEP 2944 H=2944:00 +CPU= 16.837
  12:02:18 STEP 2945 H=2945:00 +CPU= 16.460
  12:02:44 STEP 2946 H=2946:00 +CPU= 25.463
  12:03:00 STEP 2947 H=2947:00 +CPU= 16.016
  12:03:16 STEP 2948 H=2948:00 +CPU= 16.069
  12:03:41 STEP 2949 H=2949:00 +CPU= 24.426
  12:03:57 STEP 2950 H=2950:00 +CPU= 16.480
  12:04:14 STEP 2951 H=2951:00 +CPU= 16.113
  12:04:41 STEP 2952 H=2952:00 +CPU= 27.088
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.865283731608E-01 1.865286143845E-01 6.362598939121E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.260318722649E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.48989191816039E-02 -1.72994898449055E+00  1.89761310057317E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.19925486346338E-02 -1.00000000000000E+00  1.00000000000000E+00
  MAXGPFV : MAX. VALUE =    1.04859236594533     
  MAXGPFV : MAX. VALUE =   0.763544364891770     
 NSTEP =  2952 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGhnw4+002952 MODE=a
 IO-STREAM CLOSED - ICMGGhnw4+002952
 MPI-TASK:   1 -    335776826 BYTES IN      8 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0058( 5.2%)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302231500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302233000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302234500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200303000000  !!!!!!!!!!!! 
 12:04:41 STEP 2952 H=2952:00 +CPU= 27.088
 END CNT3
 NSTEP =  2952 CNT0   000000000  23.750  23.750  24.069   0.987               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           35/   804992476 WRITTEN         1119/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     30    0       35       786125     1119 ************     4.9     1.3     3.5     1.8     5.5
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1489003.5 1489003.5    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1486417.0 1486417.0     99.83      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   79    2028.1    2028.1     10.76      0.00
   9 SPCM     - SPECTRAL COMP.                      72     276.2     276.2      1.34      0.00
  10 SCAN2M - PHYSICS                               73    8575.4    8575.4     42.04      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  6     868.3     868.3      0.35      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  4      16.4      16.4      0.00      0.00
  13 SCAN2M - RADIATION CALC.                       25    9156.0    9156.0     15.37      0.00
  17 GRIDFPOS IN CNT4                                6      40.5      40.5      0.02      0.00
  27 RESTART FILES - WRITING                         3    1736.7    1736.7      0.35      0.00
  28 RESTART FILES - READING                         1    2259.9    2259.9      0.15      0.00
  29 SU4FPOS IN CNT4                                 6       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 6    1523.4    1523.4      0.61      0.00
  31 POSDDH IN STEPO                                 1       0.7       0.7      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     73    2742.4    2742.4     13.44      0.00
  38 WAM      - TOTAL COST OF WAVE MODEL            73    2020.3    2020.3      9.90      0.00
  39 SU0YOMB                                         1     207.0     207.0      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                     73      15.0      15.0      0.07      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               72     187.4     187.4      0.91      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               72      20.1      20.1      0.10      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  4       0.1       0.1      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      252     102.0     102.0      1.73      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       150      97.1      97.1      0.98      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        150      39.9      39.9      0.40      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       252      34.1      34.1      0.58      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            2      15.0      15.0      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            252      16.1      16.1      0.27      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            150      14.4      14.4      0.15      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            252      26.9      26.9      0.45      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            150      18.7      18.7      0.19      0.00
 400 GSTATS                                      15575       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 14849       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1489.0 CPU TIME    1464.9 VECTOR TIME     1464.9


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     7149.5
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhnw4+002952
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhnw4+002952
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc-client/slots/13/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/13/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMUAhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMGGhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214093/ICMSHhnw4+002928
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00i7_2019110100_123_993_12214093_0_r62653722_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
12:08:03 (589425): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:53:57 1532237 22315450 oifs_43r3_00i7_2019110100_123_993_12214093_0 10,623,600 60,222 0.0057
09 May 2023 16:53:55 1532237 22315450 oifs_43r3_00i7_2019110100_123_993_12214093_0 10,368,000 58,717 0.0057
09 May 2023 16:53:46 1532237 22315450 oifs_43r3_00i7_2019110100_123_993_12214093_0 8,812,800 49,685 0.0056
09 May 2023 16:53:40 1532237 22315450 oifs_43r3_00i7_2019110100_123_993_12214093_0 7,344,000 41,358 0.0056
09 May 2023 16:53:34 1532237 22315450 oifs_43r3_00i7_2019110100_123_993_12214093_0 6,048,000 33,659 0.0056
09 May 2023 16:53:25 1532237 22315450 oifs_43r3_00i7_2019110100_123_993_12214093_0 4,924,800 27,265 0.0055
09 May 2023 16:53:22 1532237 22315450 oifs_43r3_00i7_2019110100_123_993_12214093_0 4,060,800 22,413 0.0055
09 May 2023 16:53:14 1532237 22315450 oifs_43r3_00i7_2019110100_123_993_12214093_0 2,592,000 13,838 0.0053
09 May 2023 16:53:10 1532237 22315450 oifs_43r3_00i7_2019110100_123_993_12214093_0 1,728,000 9,119 0.0053
09 May 2023 16:53:03 1532237 22315450 oifs_43r3_00i7_2019110100_123_993_12214093_0 864,000 4,489 0.0052


©2024 climateprediction.net