climateprediction.net home page
Task 22446269

Task 22446269

Name oifs_43r3_bl_a3cd_2016092300_20_1018_12292660_0
Workunit 12292660
Created 13 Jun 2024, 2:17:58 UTC
Sent 13 Jun 2024, 14:20:30 UTC
Report deadline 12 Aug 2024, 14:20:30 UTC
Received 14 Jun 2024, 6:31:32 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1541092
Run time 14 hours 26 min 36 sec
CPU time 14 hours 26 min 36 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,023.27 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,452.20 MB
Stderr
<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
= 370:45 +CPU= 24.615
  02:17:09 STEP 1484 H= 371:00 +CPU= 25.349
  02:17:37 STEP 1485 H= 371:15 +CPU= 24.781
  02:18:04 STEP 1486 H= 371:30 +CPU= 24.863
  02:18:32 STEP 1487 H= 371:45 +CPU= 24.694
  02:19:10 STEP 1488 H= 372:00 +CPU= 35.887
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001488
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001488
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001488
  02:19:36 STEP 1489 H= 372:15 +CPU= 25.429
  02:20:03 STEP 1490 H= 372:30 +CPU= 25.544
  02:20:29 STEP 1491 H= 372:45 +CPU= 25.854
  02:20:56 STEP 1492 H= 373:00 +CPU= 26.760
  02:21:22 STEP 1493 H= 373:15 +CPU= 25.656
  02:21:49 STEP 1494 H= 373:30 +CPU= 25.978
  02:22:16 STEP 1495 H= 373:45 +CPU= 26.543
  02:22:44 STEP 1496 H= 374:00 +CPU= 26.654
  02:23:11 STEP 1497 H= 374:15 +CPU= 25.958
  02:23:36 STEP 1498 H= 374:30 +CPU= 24.316
  02:24:02 STEP 1499 H= 374:45 +CPU= 25.228
  02:24:35 STEP 1500 H= 375:00 +CPU= 32.015
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001500
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001500
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001500
  02:25:01 STEP 1501 H= 375:15 +CPU= 25.819
  02:25:27 STEP 1502 H= 375:30 +CPU= 25.586
  02:25:54 STEP 1503 H= 375:45 +CPU= 25.918
  02:26:21 STEP 1504 H= 376:00 +CPU= 26.305
  02:26:48 STEP 1505 H= 376:15 +CPU= 25.902
  02:27:15 STEP 1506 H= 376:30 +CPU= 26.449
  02:27:40 STEP 1507 H= 376:45 +CPU= 25.478
  02:28:07 STEP 1508 H= 377:00 +CPU= 26.414
  02:28:33 STEP 1509 H= 377:15 +CPU= 25.727
  02:29:00 STEP 1510 H= 377:30 +CPU= 25.931
  02:29:27 STEP 1511 H= 377:45 +CPU= 25.717
  02:29:58 STEP 1512 H= 378:00 +CPU= 31.477
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001512
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001512
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001512
  02:30:25 STEP 1513 H= 378:15 +CPU= 25.819
  02:30:52 STEP 1514 H= 378:30 +CPU= 25.921
  02:31:18 STEP 1515 H= 378:45 +CPU= 25.811
  02:31:45 STEP 1516 H= 379:00 +CPU= 26.325
  02:32:12 STEP 1517 H= 379:15 +CPU= 25.734
  02:32:38 STEP 1518 H= 379:30 +CPU= 25.765
  02:33:04 STEP 1519 H= 379:45 +CPU= 25.708
  02:33:32 STEP 1520 H= 380:00 +CPU= 26.648
  02:33:58 STEP 1521 H= 380:15 +CPU= 25.581
  02:34:24 STEP 1522 H= 380:30 +CPU= 25.738
  02:34:50 STEP 1523 H= 380:45 +CPU= 25.630
  02:35:24 STEP 1524 H= 381:00 +CPU= 32.042
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001524
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001524
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001524
  02:35:51 STEP 1525 H= 381:15 +CPU= 26.328
  02:36:18 STEP 1526 H= 381:30 +CPU= 25.720
  02:36:44 STEP 1527 H= 381:45 +CPU= 25.839
  02:37:11 STEP 1528 H= 382:00 +CPU= 26.503
  02:37:37 STEP 1529 H= 382:15 +CPU= 25.706
  02:38:04 STEP 1530 H= 382:30 +CPU= 26.116
  02:38:31 STEP 1531 H= 382:45 +CPU= 25.931
  02:38:57 STEP 1532 H= 383:00 +CPU= 26.173
  02:39:24 STEP 1533 H= 383:15 +CPU= 25.736
  02:39:50 STEP 1534 H= 383:30 +CPU= 25.827
  02:40:16 STEP 1535 H= 383:45 +CPU= 25.762
  02:40:53 STEP 1536 H= 384:00 +CPU= 35.771
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001536
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001536
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001524
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3cd_2016092300_20_1018_12292660_0_r2116832444_15.zip
Uploading the intermediate file: upload_file_15.zip
  02:41:20 STEP 1537 H= 384:15 +CPU= 25.527
  02:41:46 STEP 1538 H= 384:30 +CPU= 25.727
  02:42:12 STEP 1539 H= 384:45 +CPU= 25.898
  02:42:39 STEP 1540 H= 385:00 +CPU= 26.207
  02:43:05 STEP 1541 H= 385:15 +CPU= 25.730
  02:43:32 STEP 1542 H= 385:30 +CPU= 25.749
  02:43:58 STEP 1543 H= 385:45 +CPU= 25.910
  02:44:25 STEP 1544 H= 386:00 +CPU= 26.593
  02:44:52 STEP 1545 H= 386:15 +CPU= 25.912
  02:45:19 STEP 1546 H= 386:30 +CPU= 26.384
  02:45:45 STEP 1547 H= 386:45 +CPU= 25.733
  02:46:19 STEP 1548 H= 387:00 +CPU= 31.999
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001548
  02:46:45 STEP 1549 H= 387:15 +CPU= 25.804
  02:47:12 STEP 1550 H= 387:30 +CPU= 25.571
  02:47:38 STEP 1551 H= 387:45 +CPU= 25.767
  02:48:06 STEP 1552 H= 388:00 +CPU= 26.667
  02:48:33 STEP 1553 H= 388:15 +CPU= 26.005
  02:49:00 STEP 1554 H= 388:30 +CPU= 25.845
  02:49:27 STEP 1555 H= 388:45 +CPU= 26.205
  02:49:54 STEP 1556 H= 389:00 +CPU= 26.480
  02:50:21 STEP 1557 H= 389:15 +CPU= 26.182
  02:50:47 STEP 1558 H= 389:30 +CPU= 25.914
  02:51:14 STEP 1559 H= 389:45 +CPU= 25.917
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

 NSTEP =  1556 STEPO   0AAA00AAA
 02:49:54 STEP 1556 H= 389:00 +CPU= 26.480
 NSTEP =  1557 STEPO   0AAA00AAA
 02:50:21 STEP 1557 H= 389:15 +CPU= 26.182
 NSTEP =  1558 STEPO   0AAA00AAA
 02:50:47 STEP 1558 H= 389:30 +CPU= 25.914
 NSTEP =  1559 STEPO   0AAA00AAA
 02:51:14 STEP 1559 H= 389:45 +CPU= 25.917
ISTEP=    1560 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1560 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
  DATE= 2016    9   27
ISTEP=    1560: Reset RSOLINC= 1361.4482 RIP0= 1361.4482
 SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2:   6.175516699499999E-004
  1.087291259320000E-006  4.989444016862500E-007  3.778881179906250E-009
  2.099153872040000E-009  7.940283000000001E-011
 suecaec NAERMACC =  1
 UPDT RAESHDU =  2000.0
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20160923
         450  9.999999999999998E-005
 NORMS AT NSTEP CNT4 1560
 SPECTRAL NORMS -   LOG(PREHYDS)    0.115260934672885E+02
 LEV     VORTICITY             DIVERGENCE           TEMPERATURE           KINETIC ENERGY    
 AVE 0.489297513051412E-05 0.667588607588644E-06 0.204851352373599E+03 0.200108489413715E+02
 GPNORM  HUMIDITY            AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.213725937883922E-02 0.999999999999999E-08 0.215859399533100E-01
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.482495087587291E-06 0.000000000000000E+00 0.593213271672221E-03
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.673995496135461E-07 0.000000000000000E+00 0.764509599294170E-04
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.824879819635968E-07 0.000000000000000E+00 0.194957168824774E-03
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.153580275254170E-07 0.000000000000000E+00 0.637994921746552E-04
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.143400197591711E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.482183332073555E-06 0.000000000000000E+00 0.577660786514652E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.674036568327606E-07 0.000000000000000E+00 0.781595135712273E-04
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.143426329822508E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.690736587390207E-07 0.000000000000000E+00 0.278616445930490E-03
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.891758718086936E-07 0.000000000000000E+00 0.243080565197439E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.706631169108243E-07 0.000000000000000E+00 0.209622779082299E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.794587245302679E-08 0.000000000000000E+00 0.152611540165038E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.312526112518102E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1560 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1560 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001560 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001560
 MPI-TASK:   1 -    304703332 BYTES IN      6 RECORDS TRANSFERRED IN    0.0006 SECONDS507838.9 Mbytes/s, TOTAL TIME=    0.0092( 6.5%)
 NSTEP =  1560 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001560 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001560
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0147 SECONDS 48122.4 Mbytes/s, TOTAL TIME=    0.2070( 7.1%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001560 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001560
 MPI-TASK:   1 -   4875253312 BYTES IN    125 RECORDS TRANSFERRED IN    0.0078 SECONDS625032.5 Mbytes/s, TOTAL TIME=    0.1697( 4.6%)
 STEPO_FPOS WAS : B00000F00E
------------------------------------------------
CNT0 not found; string returned was: 'DYNFPOS'
>>> Printing last 8 lines from file: ifs.stat

 02:49:54 0AAA00AAA STEPO     1557    26.481   26.481   27.252    526:04    540:35 0.67212818412397E-06       2GB       0MB
 02:50:21 0AAA00AAA STEPO     1558    26.182   26.182   26.905    526:30    541:02 0.67212818412397E-06       2GB       0MB
 02:50:47 0AAA00AAA STEPO     1559    25.914   25.914   26.629    526:56    541:29 0.67212818412397E-06       2GB       0MB
 02:51:15 A00000000 STEPO     1560    26.067   26.067   27.112    527:22    541:56 0.66758860758864E-06       2GB       0MB
 02:51:15 0AA000000 STEPO     1560     0.024    0.024    0.024    527:22    541:56 0.66758860758864E-06       2GB       0MB
 02:51:16 FULLPOS-B DYNFPOS   1560     1.450    1.450    1.477    527:24    541:57 0.66758860758864E-06       2GB       0MB
 02:51:19 FULLPOS-V DYNFPOS   1560     2.815    2.815    2.841    527:27    542:00 0.66758860758864E-06       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1536",
 CTIME   = "00160000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1524, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc/slots/13/progress_file_12292660.xml

<running_values>
  <last_cpu_time>43387.270000</last_cpu_time>
  <upload_file_number>16</upload_file_number>
  <last_iter>1560</last_iter>
  <last_upload>1382400</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3cd
(argv4) batchid: 1018
(argv5) wuid: 12292660
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/13
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/13/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/13/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4714de73d7f5efc12a3d271f60979213 to: /var/lib/boinc/slots/13/oifs_43r3_bl_a3cd_2016092300_20_1018_12292660.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/13/oifs_43r3_bl_a3cd_2016092300_20_1018_12292660.zip
ic_ancil_file: ic_ancil_12292660
ifsdata_file: ifsdata_12292660
climate_data_file: clim_data_12292660
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/13/ic_ancil_12292660.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/13/ic_ancil_12292660.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ac062b766427427540216f6436224d2e to: /var/lib/boinc/slots/13/ifsdata/ifsdata_12292660.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/13/ifsdata/ifsdata_12292660.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/13/159l_2/clim_data_12292660.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/13/159l_2/clim_data_12292660.zip
Checking for progress XML file: /var/lib/boinc/slots/13/progress_file_12292660.xml
Opened progress file ok : /var/lib/boinc/slots/13/progress_file_12292660.xml
-- Model is restarting --
Adjusting last_iter, 1560, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/13/progress_file_12292660.xml
last_cpu_time: 43387
upload_file_number: 16
last_iter: 1537
last_upload: 1382400
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3cd_2016092300_20_1018_12292660_0_r2116832444
The child process has been launched with process id: 184
Executing the command: /var/lib/boinc/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:1c3c15f9e3f2:1:1:184:184] [20240614:025556:1718333756:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:56:24 STEP 1536 H= 384:00 +CPU= 25.377
  02:56:52 STEP 1537 H= 384:15 +CPU= 27.018
  02:57:20 STEP 1538 H= 384:30 +CPU= 27.287
  02:57:48 STEP 1539 H= 384:45 +CPU= 27.385
  02:58:18 STEP 1540 H= 385:00 +CPU= 28.958
  02:58:48 STEP 1541 H= 385:15 +CPU= 28.837
  02:59:17 STEP 1542 H= 385:30 +CPU= 28.085
  02:59:46 STEP 1543 H= 385:45 +CPU= 28.001
  03:00:16 STEP 1544 H= 386:00 +CPU= 28.467
  03:00:46 STEP 1545 H= 386:15 +CPU= 29.776
  03:01:16 STEP 1546 H= 386:30 +CPU= 28.614
  03:01:44 STEP 1547 H= 386:45 +CPU= 27.140
  03:02:20 STEP 1548 H= 387:00 +CPU= 35.202
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001548
  03:02:48 STEP 1549 H= 387:15 +CPU= 27.480
  03:03:15 STEP 1550 H= 387:30 +CPU= 26.779
  03:03:43 STEP 1551 H= 387:45 +CPU= 27.142
  03:04:13 STEP 1552 H= 388:00 +CPU= 28.675
  03:04:42 STEP 1553 H= 388:15 +CPU= 28.493
  03:05:11 STEP 1554 H= 388:30 +CPU= 29.036
  03:05:40 STEP 1555 H= 388:45 +CPU= 27.499
  03:06:09 STEP 1556 H= 389:00 +CPU= 28.434
  03:06:38 STEP 1557 H= 389:15 +CPU= 27.871
  03:07:05 STEP 1558 H= 389:30 +CPU= 26.246
  03:07:33 STEP 1559 H= 389:45 +CPU= 26.668
  03:08:07 STEP 1560 H= 390:00 +CPU= 32.734
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001560
  03:08:35 STEP 1561 H= 390:15 +CPU= 27.041
  03:09:02 STEP 1562 H= 390:30 +CPU= 26.704
  03:09:30 STEP 1563 H= 390:45 +CPU= 27.516
  03:09:59 STEP 1564 H= 391:00 +CPU= 27.919
  03:10:26 STEP 1565 H= 391:15 +CPU= 26.058
  03:10:52 STEP 1566 H= 391:30 +CPU= 25.318
  03:11:19 STEP 1567 H= 391:45 +CPU= 26.746
  03:11:50 STEP 1568 H= 392:00 +CPU= 30.195
  03:12:21 STEP 1569 H= 392:15 +CPU= 29.898
  03:12:51 STEP 1570 H= 392:30 +CPU= 27.869
  03:13:22 STEP 1571 H= 392:45 +CPU= 28.498
  03:14:00 STEP 1572 H= 393:00 +CPU= 35.657
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001572
  03:14:31 STEP 1573 H= 393:15 +CPU= 28.374
  03:15:01 STEP 1574 H= 393:30 +CPU= 28.378
  03:15:31 STEP 1575 H= 393:45 +CPU= 28.062
  03:16:03 STEP 1576 H= 394:00 +CPU= 30.103
  03:16:35 STEP 1577 H= 394:15 +CPU= 29.436
  03:17:06 STEP 1578 H= 394:30 +CPU= 28.769
  03:17:36 STEP 1579 H= 394:45 +CPU= 28.499
  03:18:07 STEP 1580 H= 395:00 +CPU= 28.962
  03:18:35 STEP 1581 H= 395:15 +CPU= 27.124
  03:19:04 STEP 1582 H= 395:30 +CPU= 26.054
  03:19:32 STEP 1583 H= 395:45 +CPU= 26.898
  03:20:14 STEP 1584 H= 396:00 +CPU= 38.541
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001584
  03:20:42 STEP 1585 H= 396:15 +CPU= 27.459
  03:21:09 STEP 1586 H= 396:30 +CPU= 26.736
  03:21:38 STEP 1587 H= 396:45 +CPU= 27.056
  03:22:06 STEP 1588 H= 397:00 +CPU= 27.746
  03:22:34 STEP 1589 H= 397:15 +CPU= 25.238
  03:23:00 STEP 1590 H= 397:30 +CPU= 25.683
  03:23:27 STEP 1591 H= 397:45 +CPU= 25.711
  03:23:55 STEP 1592 H= 398:00 +CPU= 26.663
  03:24:23 STEP 1593 H= 398:15 +CPU= 26.218
  03:24:50 STEP 1594 H= 398:30 +CPU= 25.615
  03:25:18 STEP 1595 H= 398:45 +CPU= 26.639
  03:25:51 STEP 1596 H= 399:00 +CPU= 31.350
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001596
  03:26:19 STEP 1597 H= 399:15 +CPU= 25.830
  03:26:46 STEP 1598 H= 399:30 +CPU= 25.299
  03:27:14 STEP 1599 H= 399:45 +CPU= 26.787
  03:27:43 STEP 1600 H= 400:00 +CPU= 27.229
  03:28:11 STEP 1601 H= 400:15 +CPU= 26.374
  03:28:38 STEP 1602 H= 400:30 +CPU= 25.407
  03:29:05 STEP 1603 H= 400:45 +CPU= 25.701
  03:29:33 STEP 1604 H= 401:00 +CPU= 26.127
  03:30:00 STEP 1605 H= 401:15 +CPU= 24.673
  03:30:27 STEP 1606 H= 401:30 +CPU= 24.619
  03:30:53 STEP 1607 H= 401:45 +CPU= 24.217
  03:31:27 STEP 1608 H= 402:00 +CPU= 30.223
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001608
  03:31:52 STEP 1609 H= 402:15 +CPU= 24.389
  03:32:18 STEP 1610 H= 402:30 +CPU= 24.173
  03:32:45 STEP 1611 H= 402:45 +CPU= 24.358
  03:33:12 STEP 1612 H= 403:00 +CPU= 25.080
  03:33:38 STEP 1613 H= 403:15 +CPU= 24.289
  03:34:05 STEP 1614 H= 403:30 +CPU= 24.169
  03:34:31 STEP 1615 H= 403:45 +CPU= 24.402
  03:34:58 STEP 1616 H= 404:00 +CPU= 25.077
  03:35:24 STEP 1617 H= 404:15 +CPU= 24.638
  03:35:50 STEP 1618 H= 404:30 +CPU= 24.317
  03:36:17 STEP 1619 H= 404:45 +CPU= 24.242
  03:36:49 STEP 1620 H= 405:00 +CPU= 30.037
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001620
  03:37:14 STEP 1621 H= 405:15 +CPU= 24.305
  03:37:41 STEP 1622 H= 405:30 +CPU= 24.651
  03:38:08 STEP 1623 H= 405:45 +CPU= 24.505
  03:38:35 STEP 1624 H= 406:00 +CPU= 25.005
  03:39:01 STEP 1625 H= 406:15 +CPU= 24.280
  03:39:26 STEP 1626 H= 406:30 +CPU= 24.156
  03:39:53 STEP 1627 H= 406:45 +CPU= 24.336
  03:40:19 STEP 1628 H= 407:00 +CPU= 24.762
  03:40:46 STEP 1629 H= 407:15 +CPU= 24.294
  03:41:12 STEP 1630 H= 407:30 +CPU= 24.154
  03:41:38 STEP 1631 H= 407:45 +CPU= 24.089
  03:42:16 STEP 1632 H= 408:00 +CPU= 34.166
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001620
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3cd_2016092300_20_1018_12292660_0_r2116832444_16.zip
Uploading the intermediate file: upload_file_16.zip
  03:42:42 STEP 1633 H= 408:15 +CPU= 24.156
  03:43:09 STEP 1634 H= 408:30 +CPU= 23.969
  03:43:35 STEP 1635 H= 408:45 +CPU= 24.942
  03:44:02 STEP 1636 H= 409:00 +CPU= 24.827
  03:44:29 STEP 1637 H= 409:15 +CPU= 24.514
  03:44:55 STEP 1638 H= 409:30 +CPU= 24.498
  03:45:23 STEP 1639 H= 409:45 +CPU= 24.559
  03:45:50 STEP 1640 H= 410:00 +CPU= 25.446
  03:46:17 STEP 1641 H= 410:15 +CPU= 25.031
  03:46:42 STEP 1642 H= 410:30 +CPU= 24.669
  03:47:08 STEP 1643 H= 410:45 +CPU= 24.678
  03:47:40 STEP 1644 H= 411:00 +CPU= 30.249
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001644
  03:48:06 STEP 1645 H= 411:15 +CPU= 24.211
  03:48:32 STEP 1646 H= 411:30 +CPU= 24.495
  03:48:58 STEP 1647 H= 411:45 +CPU= 24.152
  03:49:26 STEP 1648 H= 412:00 +CPU= 25.129
  03:49:52 STEP 1649 H= 412:15 +CPU= 24.929
  03:50:18 STEP 1650 H= 412:30 +CPU= 24.297
  03:50:44 STEP 1651 H= 412:45 +CPU= 24.202
  03:51:12 STEP 1652 H= 413:00 +CPU= 24.932
  03:51:38 STEP 1653 H= 413:15 +CPU= 24.412
  03:52:04 STEP 1654 H= 413:30 +CPU= 24.473
  03:52:31 STEP 1655 H= 413:45 +CPU= 24.326
  03:53:04 STEP 1656 H= 414:00 +CPU= 30.332
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001656
  03:53:30 STEP 1657 H= 414:15 +CPU= 24.143
  03:53:56 STEP 1658 H= 414:30 +CPU= 24.368
  03:54:23 STEP 1659 H= 414:45 +CPU= 24.384
  03:54:50 STEP 1660 H= 415:00 +CPU= 25.027
  03:55:15 STEP 1661 H= 415:15 +CPU= 24.298
  03:55:42 STEP 1662 H= 415:30 +CPU= 23.945
  03:56:09 STEP 1663 H= 415:45 +CPU= 24.451
  03:56:33 STEP 1664 H= 416:00 +CPU= 22.792
  03:56:54 STEP 1665 H= 416:15 +CPU= 19.723
  03:57:16 STEP 1666 H= 416:30 +CPU= 19.580
  03:57:38 STEP 1667 H= 416:45 +CPU= 19.510
  03:58:05 STEP 1668 H= 417:00 +CPU= 24.974
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001668
  03:58:25 STEP 1669 H= 417:15 +CPU= 19.692
  03:58:46 STEP 1670 H= 417:30 +CPU= 19.549
  03:59:08 STEP 1671 H= 417:45 +CPU= 19.709
  03:59:29 STEP 1672 H= 418:00 +CPU= 20.322
  03:59:50 STEP 1673 H= 418:15 +CPU= 19.366
  04:00:11 STEP 1674 H= 418:30 +CPU= 19.588
  04:00:32 STEP 1675 H= 418:45 +CPU= 19.613
  04:00:54 STEP 1676 H= 419:00 +CPU= 20.104
  04:01:15 STEP 1677 H= 419:15 +CPU= 19.564
  04:01:36 STEP 1678 H= 419:30 +CPU= 19.630
  04:01:58 STEP 1679 H= 419:45 +CPU= 19.693
  04:02:28 STEP 1680 H= 420:00 +CPU= 28.147
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001680
  04:02:49 STEP 1681 H= 420:15 +CPU= 19.763
  04:03:10 STEP 1682 H= 420:30 +CPU= 19.459
  04:03:30 STEP 1683 H= 420:45 +CPU= 19.218
  04:03:52 STEP 1684 H= 421:00 +CPU= 19.916
  04:04:12 STEP 1685 H= 421:15 +CPU= 19.521
  04:04:34 STEP 1686 H= 421:30 +CPU= 19.709
  04:04:55 STEP 1687 H= 421:45 +CPU= 19.574
  04:05:17 STEP 1688 H= 422:00 +CPU= 20.205
  04:05:38 STEP 1689 H= 422:15 +CPU= 19.519
  04:05:59 STEP 1690 H= 422:30 +CPU= 19.143
  04:06:20 STEP 1691 H= 422:45 +CPU= 19.708
  04:06:47 STEP 1692 H= 423:00 +CPU= 25.011
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001692
  04:07:09 STEP 1693 H= 423:15 +CPU= 19.594
  04:07:30 STEP 1694 H= 423:30 +CPU= 19.591
  04:07:50 STEP 1695 H= 423:45 +CPU= 19.461
  04:08:12 STEP 1696 H= 424:00 +CPU= 19.916
  04:08:32 STEP 1697 H= 424:15 +CPU= 19.508
  04:08:54 STEP 1698 H= 424:30 +CPU= 19.427
  04:09:15 STEP 1699 H= 424:45 +CPU= 19.733
  04:09:37 STEP 1700 H= 425:00 +CPU= 19.920
  04:09:57 STEP 1701 H= 425:15 +CPU= 19.588
  04:10:18 STEP 1702 H= 425:30 +CPU= 19.543
  04:10:39 STEP 1703 H= 425:45 +CPU= 19.551
  04:11:06 STEP 1704 H= 426:00 +CPU= 24.917
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001704
  04:11:27 STEP 1705 H= 426:15 +CPU= 19.668
  04:11:48 STEP 1706 H= 426:30 +CPU= 19.826
  04:12:10 STEP 1707 H= 426:45 +CPU= 19.462
  04:12:31 STEP 1708 H= 427:00 +CPU= 19.781
  04:12:51 STEP 1709 H= 427:15 +CPU= 19.472
  04:13:13 STEP 1710 H= 427:30 +CPU= 19.669
  04:13:34 STEP 1711 H= 427:45 +CPU= 19.758
  04:13:56 STEP 1712 H= 428:00 +CPU= 20.450
  04:14:16 STEP 1713 H= 428:15 +CPU= 19.423
  04:14:38 STEP 1714 H= 428:30 +CPU= 19.403
  04:14:59 STEP 1715 H= 428:45 +CPU= 19.621
  04:15:26 STEP 1716 H= 429:00 +CPU= 24.795
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001716
  04:15:47 STEP 1717 H= 429:15 +CPU= 19.841
  04:16:06 STEP 1718 H= 429:30 +CPU= 19.311
  04:16:27 STEP 1719 H= 429:45 +CPU= 19.503
  04:16:49 STEP 1720 H= 430:00 +CPU= 20.033
  04:17:11 STEP 1721 H= 430:15 +CPU= 20.061
  04:17:31 STEP 1722 H= 430:30 +CPU= 19.201
  04:17:53 STEP 1723 H= 430:45 +CPU= 19.908
  04:18:14 STEP 1724 H= 431:00 +CPU= 20.227
  04:18:36 STEP 1725 H= 431:15 +CPU= 19.727
  04:18:56 STEP 1726 H= 431:30 +CPU= 19.506
  04:19:18 STEP 1727 H= 431:45 +CPU= 20.068
  04:19:48 STEP 1728 H= 432:00 +CPU= 28.277
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001716
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3cd_2016092300_20_1018_12292660_0_r2116832444_17.zip
Uploading the intermediate file: upload_file_17.zip
  04:20:09 STEP 1729 H= 432:15 +CPU= 19.327
  04:20:29 STEP 1730 H= 432:30 +CPU= 19.796
  04:20:52 STEP 1731 H= 432:45 +CPU= 19.610
  04:21:13 STEP 1732 H= 433:00 +CPU= 20.118
  04:21:34 STEP 1733 H= 433:15 +CPU= 19.213
  04:21:55 STEP 1734 H= 433:30 +CPU= 19.389
  04:22:16 STEP 1735 H= 433:45 +CPU= 19.719
  04:22:38 STEP 1736 H= 434:00 +CPU= 20.232
  04:22:59 STEP 1737 H= 434:15 +CPU= 19.471
  04:23:20 STEP 1738 H= 434:30 +CPU= 19.537
  04:23:40 STEP 1739 H= 434:45 +CPU= 19.536
  04:24:07 STEP 1740 H= 435:00 +CPU= 24.723
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001740
  04:24:27 STEP 1741 H= 435:15 +CPU= 19.119
  04:24:48 STEP 1742 H= 435:30 +CPU= 19.755
  04:25:09 STEP 1743 H= 435:45 +CPU= 19.793
  04:25:31 STEP 1744 H= 436:00 +CPU= 20.032
  04:25:52 STEP 1745 H= 436:15 +CPU= 19.545
  04:26:14 STEP 1746 H= 436:30 +CPU= 19.826
  04:26:34 STEP 1747 H= 436:45 +CPU= 19.524
  04:26:56 STEP 1748 H= 437:00 +CPU= 20.515
  04:27:17 STEP 1749 H= 437:15 +CPU= 19.794
  04:27:38 STEP 1750 H= 437:30 +CPU= 19.221
  04:27:58 STEP 1751 H= 437:45 +CPU= 19.565
  04:28:25 STEP 1752 H= 438:00 +CPU= 25.116
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001752
  04:28:45 STEP 1753 H= 438:15 +CPU= 19.686
  04:29:06 STEP 1754 H= 438:30 +CPU= 19.405
  04:29:27 STEP 1755 H= 438:45 +CPU= 19.643
  04:29:49 STEP 1756 H= 439:00 +CPU= 20.134
  04:30:10 STEP 1757 H= 439:15 +CPU= 19.472
  04:30:30 STEP 1758 H= 439:30 +CPU= 19.457
  04:30:51 STEP 1759 H= 439:45 +CPU= 19.460
  04:31:13 STEP 1760 H= 440:00 +CPU= 20.625
  04:31:34 STEP 1761 H= 440:15 +CPU= 19.542
  04:31:55 STEP 1762 H= 440:30 +CPU= 19.591
  04:32:17 STEP 1763 H= 440:45 +CPU= 19.818
  04:32:44 STEP 1764 H= 441:00 +CPU= 24.799
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001764
  04:33:05 STEP 1765 H= 441:15 +CPU= 19.752
  04:33:27 STEP 1766 H= 441:30 +CPU= 19.843
  04:33:48 STEP 1767 H= 441:45 +CPU= 19.762
  04:34:09 STEP 1768 H= 442:00 +CPU= 20.288
  04:34:31 STEP 1769 H= 442:15 +CPU= 19.944
  04:34:52 STEP 1770 H= 442:30 +CPU= 19.588
  04:35:13 STEP 1771 H= 442:45 +CPU= 19.309
  04:35:34 STEP 1772 H= 443:00 +CPU= 20.178
  04:35:55 STEP 1773 H= 443:15 +CPU= 19.534
  04:36:17 STEP 1774 H= 443:30 +CPU= 19.631
  04:36:38 STEP 1775 H= 443:45 +CPU= 19.829
  04:37:08 STEP 1776 H= 444:00 +CPU= 28.210
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001776
  04:37:29 STEP 1777 H= 444:15 +CPU= 20.061
  04:37:50 STEP 1778 H= 444:30 +CPU= 19.726
  04:38:11 STEP 1779 H= 444:45 +CPU= 19.444
  04:38:33 STEP 1780 H= 445:00 +CPU= 20.243
  04:38:55 STEP 1781 H= 445:15 +CPU= 19.650
  04:39:15 STEP 1782 H= 445:30 +CPU= 19.453
  04:39:36 STEP 1783 H= 445:45 +CPU= 19.670
  04:39:57 STEP 1784 H= 446:00 +CPU= 20.246
  04:40:18 STEP 1785 H= 446:15 +CPU= 19.555
  04:40:39 STEP 1786 H= 446:30 +CPU= 19.476
  04:41:01 STEP 1787 H= 446:45 +CPU= 19.978
  04:41:26 STEP 1788 H= 447:00 +CPU= 24.398
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001788
  04:41:48 STEP 1789 H= 447:15 +CPU= 19.448
  04:42:08 STEP 1790 H= 447:30 +CPU= 19.481
  04:42:30 STEP 1791 H= 447:45 +CPU= 19.603
  04:42:51 STEP 1792 H= 448:00 +CPU= 20.298
  04:43:12 STEP 1793 H= 448:15 +CPU= 19.454
  04:43:33 STEP 1794 H= 448:30 +CPU= 19.372
  04:43:54 STEP 1795 H= 448:45 +CPU= 19.535
  04:44:16 STEP 1796 H= 449:00 +CPU= 20.580
  04:44:37 STEP 1797 H= 449:15 +CPU= 19.590
  04:44:58 STEP 1798 H= 449:30 +CPU= 19.785
  04:45:19 STEP 1799 H= 449:45 +CPU= 19.358
  04:45:46 STEP 1800 H= 450:00 +CPU= 25.242
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001800
  04:46:07 STEP 1801 H= 450:15 +CPU= 19.380
  04:46:28 STEP 1802 H= 450:30 +CPU= 19.461
  04:46:49 STEP 1803 H= 450:45 +CPU= 19.823
  04:47:10 STEP 1804 H= 451:00 +CPU= 20.150
  04:47:31 STEP 1805 H= 451:15 +CPU= 19.603
  04:47:52 STEP 1806 H= 451:30 +CPU= 19.633
  04:48:13 STEP 1807 H= 451:45 +CPU= 19.450
  04:48:35 STEP 1808 H= 452:00 +CPU= 20.480
  04:48:56 STEP 1809 H= 452:15 +CPU= 19.868
  04:49:17 STEP 1810 H= 452:30 +CPU= 19.311
  04:49:38 STEP 1811 H= 452:45 +CPU= 19.722
  04:50:04 STEP 1812 H= 453:00 +CPU= 24.596
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001812
  04:50:25 STEP 1813 H= 453:15 +CPU= 19.492
  04:50:45 STEP 1814 H= 453:30 +CPU= 19.571
  04:51:06 STEP 1815 H= 453:45 +CPU= 19.474
  04:51:28 STEP 1816 H= 454:00 +CPU= 20.050
  04:51:49 STEP 1817 H= 454:15 +CPU= 19.444
  04:52:10 STEP 1818 H= 454:30 +CPU= 19.634
  04:52:30 STEP 1819 H= 454:45 +CPU= 19.411
  04:52:52 STEP 1820 H= 455:00 +CPU= 20.127
  04:53:13 STEP 1821 H= 455:15 +CPU= 19.687
  04:53:34 STEP 1822 H= 455:30 +CPU= 19.726
  04:53:54 STEP 1823 H= 455:45 +CPU= 19.432
  04:54:25 STEP 1824 H= 456:00 +CPU= 28.357
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3cd_2016092300_20_1018_12292660_0_r2116832444_18.zip
Uploading the intermediate file: upload_file_18.zip
  04:54:45 STEP 1825 H= 456:15 +CPU= 19.362
  04:55:05 STEP 1826 H= 456:30 +CPU= 19.851
  04:55:26 STEP 1827 H= 456:45 +CPU= 20.024
  04:55:48 STEP 1828 H= 457:00 +CPU= 20.069
  04:56:09 STEP 1829 H= 457:15 +CPU= 19.535
  04:56:30 STEP 1830 H= 457:30 +CPU= 19.670
  04:56:51 STEP 1831 H= 457:45 +CPU= 19.790
  04:57:13 STEP 1832 H= 458:00 +CPU= 20.328
  04:57:34 STEP 1833 H= 458:15 +CPU= 19.599
  04:57:56 STEP 1834 H= 458:30 +CPU= 19.662
  04:58:16 STEP 1835 H= 458:45 +CPU= 19.434
  04:58:43 STEP 1836 H= 459:00 +CPU= 25.104
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001836
  04:59:04 STEP 1837 H= 459:15 +CPU= 19.725
  04:59:25 STEP 1838 H= 459:30 +CPU= 19.547
  04:59:46 STEP 1839 H= 459:45 +CPU= 19.603
  05:00:08 STEP 1840 H= 460:00 +CPU= 20.341
  05:00:29 STEP 1841 H= 460:15 +CPU= 19.602
  05:00:50 STEP 1842 H= 460:30 +CPU= 19.285
  05:01:11 STEP 1843 H= 460:45 +CPU= 19.636
  05:01:33 STEP 1844 H= 461:00 +CPU= 20.329
  05:01:54 STEP 1845 H= 461:15 +CPU= 19.823
  05:02:16 STEP 1846 H= 461:30 +CPU= 19.457
  05:02:37 STEP 1847 H= 461:45 +CPU= 19.970
  05:03:04 STEP 1848 H= 462:00 +CPU= 25.218
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001848
  05:03:25 STEP 1849 H= 462:15 +CPU= 19.689
  05:03:46 STEP 1850 H= 462:30 +CPU= 19.785
  05:04:07 STEP 1851 H= 462:45 +CPU= 19.707
  05:04:29 STEP 1852 H= 463:00 +CPU= 20.205
  05:04:50 STEP 1853 H= 463:15 +CPU= 19.861
  05:05:12 STEP 1854 H= 463:30 +CPU= 19.768
  05:05:32 STEP 1855 H= 463:45 +CPU= 19.336
  05:05:54 STEP 1856 H= 464:00 +CPU= 20.156
  05:06:16 STEP 1857 H= 464:15 +CPU= 19.774
  05:06:37 STEP 1858 H= 464:30 +CPU= 19.936
  05:06:58 STEP 1859 H= 464:45 +CPU= 19.691
  05:07:25 STEP 1860 H= 465:00 +CPU= 24.783
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001860
  05:07:45 STEP 1861 H= 465:15 +CPU= 19.333
  05:08:07 STEP 1862 H= 465:30 +CPU= 19.682
  05:08:27 STEP 1863 H= 465:45 +CPU= 19.700
  05:08:49 STEP 1864 H= 466:00 +CPU= 19.863
  05:09:10 STEP 1865 H= 466:15 +CPU= 19.668
  05:09:31 STEP 1866 H= 466:30 +CPU= 19.800
  05:09:52 STEP 1867 H= 466:45 +CPU= 19.542
  05:10:14 STEP 1868 H= 467:00 +CPU= 20.189
  05:10:35 STEP 1869 H= 467:15 +CPU= 19.189
  05:10:55 STEP 1870 H= 467:30 +CPU= 19.477
  05:11:17 STEP 1871 H= 467:45 +CPU= 19.747
  05:11:47 STEP 1872 H= 468:00 +CPU= 28.251
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001872
  05:12:08 STEP 1873 H= 468:15 +CPU= 19.802
  05:12:30 STEP 1874 H= 468:30 +CPU= 19.761
  05:12:50 STEP 1875 H= 468:45 +CPU= 19.565
  05:13:12 STEP 1876 H= 469:00 +CPU= 20.060
  05:13:33 STEP 1877 H= 469:15 +CPU= 19.748
  05:13:55 STEP 1878 H= 469:30 +CPU= 19.733
  05:14:16 STEP 1879 H= 469:45 +CPU= 19.518
  05:14:36 STEP 1880 H= 470:00 +CPU= 20.041
  05:14:57 STEP 1881 H= 470:15 +CPU= 19.808
  05:15:18 STEP 1882 H= 470:30 +CPU= 19.660
  05:15:40 STEP 1883 H= 470:45 +CPU= 19.525
  05:16:07 STEP 1884 H= 471:00 +CPU= 24.924
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001884
  05:16:27 STEP 1885 H= 471:15 +CPU= 19.579
  05:16:48 STEP 1886 H= 471:30 +CPU= 19.678
  05:17:09 STEP 1887 H= 471:45 +CPU= 19.471
  05:17:31 STEP 1888 H= 472:00 +CPU= 20.052
  05:17:53 STEP 1889 H= 472:15 +CPU= 19.801
  05:18:14 STEP 1890 H= 472:30 +CPU= 19.418
  05:18:35 STEP 1891 H= 472:45 +CPU= 19.636
  05:18:56 STEP 1892 H= 473:00 +CPU= 19.663
  05:19:18 STEP 1893 H= 473:15 +CPU= 19.484
  05:19:39 STEP 1894 H= 473:30 +CPU= 20.078
  05:19:59 STEP 1895 H= 473:45 +CPU= 19.381
  05:20:26 STEP 1896 H= 474:00 +CPU= 25.182
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001896
  05:20:47 STEP 1897 H= 474:15 +CPU= 19.476
  05:21:08 STEP 1898 H= 474:30 +CPU= 19.598
  05:21:29 STEP 1899 H= 474:45 +CPU= 19.605
  05:21:51 STEP 1900 H= 475:00 +CPU= 20.209
  05:22:11 STEP 1901 H= 475:15 +CPU= 19.799
  05:22:33 STEP 1902 H= 475:30 +CPU= 19.747
  05:22:53 STEP 1903 H= 475:45 +CPU= 19.638
  05:23:16 STEP 1904 H= 476:00 +CPU= 20.377
  05:23:36 STEP 1905 H= 476:15 +CPU= 19.522
  05:23:57 STEP 1906 H= 476:30 +CPU= 19.601
  05:24:18 STEP 1907 H= 476:45 +CPU= 19.559
  05:24:45 STEP 1908 H= 477:00 +CPU= 25.013
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001908
  05:25:07 STEP 1909 H= 477:15 +CPU= 20.124
  05:25:28 STEP 1910 H= 477:30 +CPU= 19.602
  05:25:49 STEP 1911 H= 477:45 +CPU= 19.402
  05:26:11 STEP 1912 H= 478:00 +CPU= 20.050
  05:26:32 STEP 1913 H= 478:15 +CPU= 19.928
  05:26:53 STEP 1914 H= 478:30 +CPU= 19.878
  05:27:14 STEP 1915 H= 478:45 +CPU= 19.674
  05:27:35 STEP 1916 H= 479:00 +CPU= 20.390
  05:27:56 STEP 1917 H= 479:15 +CPU= 19.604
  05:28:16 STEP 1918 H= 479:30 +CPU= 19.828
  05:28:37 STEP 1919 H= 479:45 +CPU= 19.830
  05:29:06 STEP 1920 H= 480:00 +CPU= 26.428
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1250388199E-20      I       0.6977045125E-06      I
ICloud fractionI      0.0000000000E+00      I       0.3263251822E-01      I
I  Relat. Hum. I      0.3833876227E+00      I       0.4391661901E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.2668247896E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    300709732 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS501182.9 Mbytes/s, TOTAL TIME=    0.0124( 4.8%)
 05:29:06 STEP 1920 H= 480:00 +CPU= 26.428
 END CNT3
 NSTEP =  1920 CNT0   000000000  18.826  18.826  22.090   0.852               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        11440/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    203    0       46      1044068    11440 ************    24.1     1.0    23.1     1.6    43.8
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 9191611.7 9191611.7    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 9187753.2 9187753.2     99.96      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  417    4174.3    4174.3     18.94      0.00
   9 SPCM     - SPECTRAL COMP.                     384     636.7     636.7      2.66      0.00
  10 SCAN2M - PHYSICS                              385   10775.5   10775.5     45.13      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 32     881.0     881.0      0.31      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 17      49.0      49.0      0.01      0.00
  17 GRIDFPOS IN CNT4                               32      24.2      24.2      0.01      0.00
  27 RESTART FILES - WRITING                         8    3523.8    3523.8      0.31      0.00
  28 RESTART FILES - READING                         1    2240.1    2240.1      0.02      0.00
  29 SU4FPOS IN CNT4                                32       0.0       0.0      0.00      0.00
  30 DYNFPOS IN CNT4                                32    5261.9    5261.9      1.83      0.00
  31 POSDDH IN STEPO                                 2     879.7     879.7      0.02      0.00
  37 CPGLAG   - SL COMPUTATIONS                    385    5101.4    5101.4     21.37      0.00
  39 SU0YOMB                                         1     357.8     357.8      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    385      21.1      21.1      0.09      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              384     258.4     258.4      1.08      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              384      52.5      52.5      0.22      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 17       0.7       0.7      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      836     342.0     342.0      3.11      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       800     208.6     208.6      1.82      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        800      83.3      83.3      0.72      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       836     110.0     110.0      1.00      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      28.4      28.4      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            836      59.8      59.8      0.54      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            800      30.3      30.3      0.26      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            836      67.6      67.6      0.61      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            800      38.4      38.4      0.33      0.00
 400 GSTATS                                      45508       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 42288       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     9191.6 CPU TIME    8609.3 VECTOR TIME     8609.3


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      188.1
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc/slots/13/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/13/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292660/ICMGGh7zg+001860
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3cd_2016092300_20_1018_12292660_0_r2116832444_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
05:32:43 (159): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
14 Jun 2024 05:31:16 1541092 22446269 oifs_43r3_bl_a3cd_2016092300_20_1018_12292660_0 1,727,100 51,996 0.0301
14 Jun 2024 01:10:52 1541092 22446269 oifs_43r3_bl_a3cd_2016092300_20_1018_12292660_0 1,209,600 37,538 0.0310


©2024 cpdn.org