climateprediction.net home page
Task 22435809

Task 22435809

Name oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0
Workunit 12284229
Created 7 Jun 2024, 14:19:12 UTC
Sent 7 Jun 2024, 14:30:13 UTC
Report deadline 6 Aug 2024, 14:30:13 UTC
Received 9 Jun 2024, 16:21:50 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1523278
Run time 10 hours 17 min 50 sec
CPU time 10 hours 17 min 50 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 8.64 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 4,999.43 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,589.42 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
TEP 1036 H= 259:00 +CPU= 24.829
  14:11:19 STEP 1037 H= 259:15 +CPU= 24.482
  14:11:45 STEP 1038 H= 259:30 +CPU= 24.265
  14:12:09 STEP 1039 H= 259:45 +CPU= 24.380
  14:12:34 STEP 1040 H= 260:00 +CPU= 25.825
  14:12:59 STEP 1041 H= 260:15 +CPU= 24.815
  14:13:23 STEP 1042 H= 260:30 +CPU= 24.413
  14:13:48 STEP 1043 H= 260:45 +CPU= 25.125
  14:14:18 STEP 1044 H= 261:00 +CPU= 28.893
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001044
  14:14:42 STEP 1045 H= 261:15 +CPU= 24.934
  14:15:08 STEP 1046 H= 261:30 +CPU= 24.900
  14:15:32 STEP 1047 H= 261:45 +CPU= 24.670
  14:15:57 STEP 1048 H= 262:00 +CPU= 25.250
  14:16:23 STEP 1049 H= 262:15 +CPU= 25.603
  14:16:51 STEP 1050 H= 262:30 +CPU= 27.502
  14:17:19 STEP 1051 H= 262:45 +CPU= 27.452
  14:17:48 STEP 1052 H= 263:00 +CPU= 29.174
  14:18:16 STEP 1053 H= 263:15 +CPU= 27.838
  14:18:41 STEP 1054 H= 263:30 +CPU= 25.439
  14:19:04 STEP 1055 H= 263:45 +CPU= 23.944
  14:19:40 STEP 1056 H= 264:00 +CPU= 35.845
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0_r894036332_10.zip
Uploading the intermediate file: upload_file_10.zip
  14:20:04 STEP 1057 H= 264:15 +CPU= 24.425
  14:20:29 STEP 1058 H= 264:30 +CPU= 24.508
  14:20:56 STEP 1059 H= 264:45 +CPU= 25.719
  14:21:22 STEP 1060 H= 265:00 +CPU= 25.810
  14:21:47 STEP 1061 H= 265:15 +CPU= 24.979
  14:22:12 STEP 1062 H= 265:30 +CPU= 25.649
  14:22:38 STEP 1063 H= 265:45 +CPU= 27.465
  14:23:07 STEP 1064 H= 266:00 +CPU= 28.061
  14:23:34 STEP 1065 H= 266:15 +CPU= 27.668
  14:24:02 STEP 1066 H= 266:30 +CPU= 27.410
  14:24:30 STEP 1067 H= 266:45 +CPU= 27.447
  14:25:04 STEP 1068 H= 267:00 +CPU= 34.252
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001068
  14:25:31 STEP 1069 H= 267:15 +CPU= 27.568
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.879] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 5751MB (heap), 5021MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.880] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.880] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.880] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.880] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.880] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 5751 MB (maxheap), 5021 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     1        (1004) 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               LAPINEA 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                LARMES 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 LARCINA 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142541:1717849541:78912.881] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  LASCAW 
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142542:1717849542:78913.882] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94103:94103] [20240608:142542:1717849542:78913.882] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000D7437B  lascaw_                   496  lascaw.F90
oifs_43r3_model.e  0000000000D7F7D5  larcina_                  319  larcina.F90
oifs_43r3_model.e  0000000000D86678  larmes_                   603  larmes.F90
oifs_43r3_model.e  0000000000D5968D  lapinea_                  270  lapinea.F90
oifs_43r3_model.e  0000000000D56B78  call_sl_                  300  call_sl.F90
oifs_43r3_model.e  0000000000D54BBC  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BD4FD8  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1uq
(argv4) batchid: 1017
(argv5) wuid: 12284229
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ba4afbba1761108b2c22833f9c9823c1 to: /var/lib/boinc-client/slots/3/oifs_43r3_bl_a1uq_2016092300_20_1017_12284229.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_bl_a1uq_2016092300_20_1017_12284229.zip
ic_ancil_file: ic_ancil_12284229
ifsdata_file: ifsdata_12284229
climate_data_file: clim_data_12284229
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/3/ic_ancil_12284229.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12284229.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2a7faefd8326f6f83bcf9fc33ef28d0b to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12284229.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12284229.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12284229.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12284229.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12284229.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12284229.xml
-- Model is restarting --
Adjusting last_iter, 1070, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12284229.xml
last_cpu_time: 28487
upload_file_number: 11
last_iter: 1057
last_upload: 950400
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0_r894036332
The child process has been launched with process id: 5482
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5482:5482] [20240609:080002:1717912802:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:00:16 STEP 1056 H= 264:00 +CPU= 11.358
  08:00:26 STEP 1057 H= 264:15 +CPU= 10.596
  08:00:37 STEP 1058 H= 264:30 +CPU= 10.573
  08:00:48 STEP 1059 H= 264:45 +CPU= 10.575
  08:00:59 STEP 1060 H= 265:00 +CPU= 10.969
  08:01:09 STEP 1061 H= 265:15 +CPU= 10.596
  08:01:20 STEP 1062 H= 265:30 +CPU= 10.603
  08:01:30 STEP 1063 H= 265:45 +CPU= 10.610
  08:01:41 STEP 1064 H= 266:00 +CPU= 10.971
  08:01:52 STEP 1065 H= 266:15 +CPU= 10.578
  08:02:03 STEP 1066 H= 266:30 +CPU= 10.672
  08:02:13 STEP 1067 H= 266:45 +CPU= 10.682
  08:02:27 STEP 1068 H= 267:00 +CPU= 14.046
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001068
  08:02:38 STEP 1069 H= 267:15 +CPU= 10.652
  08:02:49 STEP 1070 H= 267:30 +CPU= 10.674
  08:02:59 STEP 1071 H= 267:45 +CPU= 10.721
  08:03:11 STEP 1072 H= 268:00 +CPU= 11.115
  08:03:21 STEP 1073 H= 268:15 +CPU= 10.835
  08:03:32 STEP 1074 H= 268:30 +CPU= 10.792
  08:03:43 STEP 1075 H= 268:45 +CPU= 10.698
  08:03:54 STEP 1076 H= 269:00 +CPU= 11.135
  08:04:05 STEP 1077 H= 269:15 +CPU= 10.812
  08:04:16 STEP 1078 H= 269:30 +CPU= 10.775
  08:04:27 STEP 1079 H= 269:45 +CPU= 11.005
  08:04:41 STEP 1080 H= 270:00 +CPU= 14.504
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001080
  08:04:52 STEP 1081 H= 270:15 +CPU= 11.033
  08:05:04 STEP 1082 H= 270:30 +CPU= 11.524
  08:05:15 STEP 1083 H= 270:45 +CPU= 11.199
  08:05:27 STEP 1084 H= 271:00 +CPU= 11.767
  08:05:38 STEP 1085 H= 271:15 +CPU= 11.680
  08:05:50 STEP 1086 H= 271:30 +CPU= 11.320
  08:06:01 STEP 1087 H= 271:45 +CPU= 11.287
  08:06:13 STEP 1088 H= 272:00 +CPU= 11.771
  08:06:24 STEP 1089 H= 272:15 +CPU= 11.350
  08:06:35 STEP 1090 H= 272:30 +CPU= 11.137
  08:06:47 STEP 1091 H= 272:45 +CPU= 11.250
  08:07:01 STEP 1092 H= 273:00 +CPU= 14.736
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001092
  08:07:13 STEP 1093 H= 273:15 +CPU= 11.330
  08:07:24 STEP 1094 H= 273:30 +CPU= 11.198
  08:07:35 STEP 1095 H= 273:45 +CPU= 11.320
  08:07:47 STEP 1096 H= 274:00 +CPU= 11.834
  08:07:59 STEP 1097 H= 274:15 +CPU= 11.480
  08:08:10 STEP 1098 H= 274:30 +CPU= 11.425
  08:08:21 STEP 1099 H= 274:45 +CPU= 11.437
  08:08:33 STEP 1100 H= 275:00 +CPU= 11.801
  08:08:45 STEP 1101 H= 275:15 +CPU= 11.503
  08:08:56 STEP 1102 H= 275:30 +CPU= 11.428
  08:09:08 STEP 1103 H= 275:45 +CPU= 11.485
  08:09:25 STEP 1104 H= 276:00 +CPU= 17.575
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001104
  08:09:36 STEP 1105 H= 276:15 +CPU= 11.234
  08:09:47 STEP 1106 H= 276:30 +CPU= 11.098
  08:09:59 STEP 1107 H= 276:45 +CPU= 11.173
  08:10:11 STEP 1108 H= 277:00 +CPU= 11.959
  08:10:22 STEP 1109 H= 277:15 +CPU= 11.557
  08:10:34 STEP 1110 H= 277:30 +CPU= 11.923
  08:10:46 STEP 1111 H= 277:45 +CPU= 11.980
  08:10:58 STEP 1112 H= 278:00 +CPU= 12.255
  08:11:11 STEP 1113 H= 278:15 +CPU= 12.318
  08:11:23 STEP 1114 H= 278:30 +CPU= 12.592
  08:11:35 STEP 1115 H= 278:45 +CPU= 12.176
  08:11:51 STEP 1116 H= 279:00 +CPU= 16.057
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001116
  08:12:04 STEP 1117 H= 279:15 +CPU= 12.292
  08:12:16 STEP 1118 H= 279:30 +CPU= 12.769
  08:12:29 STEP 1119 H= 279:45 +CPU= 12.401
  08:12:42 STEP 1120 H= 280:00 +CPU= 12.983
  08:12:54 STEP 1121 H= 280:15 +CPU= 12.556
  08:13:07 STEP 1122 H= 280:30 +CPU= 12.362
  08:13:19 STEP 1123 H= 280:45 +CPU= 12.703
  08:13:33 STEP 1124 H= 281:00 +CPU= 13.155
  08:13:45 STEP 1125 H= 281:15 +CPU= 12.552
  08:13:59 STEP 1126 H= 281:30 +CPU= 12.661
  08:14:12 STEP 1127 H= 281:45 +CPU= 13.195
  08:14:29 STEP 1128 H= 282:00 +CPU= 17.271
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001128
  08:14:43 STEP 1129 H= 282:15 +CPU= 13.995
  08:14:57 STEP 1130 H= 282:30 +CPU= 14.491
  08:15:12 STEP 1131 H= 282:45 +CPU= 14.465
  08:15:28 STEP 1132 H= 283:00 +CPU= 15.743
  08:15:45 STEP 1133 H= 283:15 +CPU= 16.229
  08:16:00 STEP 1134 H= 283:30 +CPU= 15.661
  08:16:16 STEP 1135 H= 283:45 +CPU= 16.345
  08:16:33 STEP 1136 H= 284:00 +CPU= 17.449
  08:16:52 STEP 1137 H= 284:15 +CPU= 18.726
  08:17:10 STEP 1138 H= 284:30 +CPU= 18.076
  08:17:31 STEP 1139 H= 284:45 +CPU= 20.067
  08:17:55 STEP 1140 H= 285:00 +CPU= 24.676
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001140
  08:18:17 STEP 1141 H= 285:15 +CPU= 22.294
  08:18:40 STEP 1142 H= 285:30 +CPU= 23.025
  08:19:05 STEP 1143 H= 285:45 +CPU= 24.743
  08:19:31 STEP 1144 H= 286:00 +CPU= 25.764
  08:19:54 STEP 1145 H= 286:15 +CPU= 23.530
  08:20:20 STEP 1146 H= 286:30 +CPU= 26.039
  08:20:46 STEP 1147 H= 286:45 +CPU= 25.156
  08:21:12 STEP 1148 H= 287:00 +CPU= 26.480
  08:21:37 STEP 1149 H= 287:15 +CPU= 25.365
  08:22:03 STEP 1150 H= 287:30 +CPU= 26.294
  08:22:27 STEP 1151 H= 287:45 +CPU= 24.248
  08:23:03 STEP 1152 H= 288:00 +CPU= 35.303
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0_r894036332_11.zip
Uploading the intermediate file: upload_file_11.zip
  08:23:27 STEP 1153 H= 288:15 +CPU= 23.907
  08:23:53 STEP 1154 H= 288:30 +CPU= 25.975
  08:24:20 STEP 1155 H= 288:45 +CPU= 27.149
  08:24:46 STEP 1156 H= 289:00 +CPU= 26.136
  08:25:11 STEP 1157 H= 289:15 +CPU= 24.559
  08:25:36 STEP 1158 H= 289:30 +CPU= 25.544
  08:26:02 STEP 1159 H= 289:45 +CPU= 25.484
  08:26:27 STEP 1160 H= 290:00 +CPU= 25.446
  08:26:53 STEP 1161 H= 290:15 +CPU= 25.213
  08:27:18 STEP 1162 H= 290:30 +CPU= 25.415
  08:27:42 STEP 1163 H= 290:45 +CPU= 24.083
  08:28:12 STEP 1164 H= 291:00 +CPU= 29.881
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001164
  08:28:36 STEP 1165 H= 291:15 +CPU= 24.226
  08:29:00 STEP 1166 H= 291:30 +CPU= 24.026
  08:29:24 STEP 1167 H= 291:45 +CPU= 24.048
  08:29:49 STEP 1168 H= 292:00 +CPU= 24.580
  08:30:13 STEP 1169 H= 292:15 +CPU= 24.570
  08:30:38 STEP 1170 H= 292:30 +CPU= 24.453
  08:31:02 STEP 1171 H= 292:45 +CPU= 24.171
  08:31:27 STEP 1172 H= 293:00 +CPU= 25.386
  08:31:52 STEP 1173 H= 293:15 +CPU= 24.686
  08:32:16 STEP 1174 H= 293:30 +CPU= 24.331
  08:32:40 STEP 1175 H= 293:45 +CPU= 23.542
  08:33:10 STEP 1176 H= 294:00 +CPU= 29.706
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001176
  08:33:34 STEP 1177 H= 294:15 +CPU= 24.009
  08:33:58 STEP 1178 H= 294:30 +CPU= 24.574
  08:34:22 STEP 1179 H= 294:45 +CPU= 23.636
  08:34:47 STEP 1180 H= 295:00 +CPU= 24.586
  08:35:10 STEP 1181 H= 295:15 +CPU= 23.571
  08:35:36 STEP 1182 H= 295:30 +CPU= 25.364
  08:36:01 STEP 1183 H= 295:45 +CPU= 25.256
  08:36:25 STEP 1184 H= 296:00 +CPU= 24.397
  08:36:50 STEP 1185 H= 296:15 +CPU= 24.922
  08:37:15 STEP 1186 H= 296:30 +CPU= 24.350
  08:37:38 STEP 1187 H= 296:45 +CPU= 23.543
  08:38:09 STEP 1188 H= 297:00 +CPU= 30.973
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001188
  08:38:33 STEP 1189 H= 297:15 +CPU= 24.296
  08:38:58 STEP 1190 H= 297:30 +CPU= 24.510
  08:39:22 STEP 1191 H= 297:45 +CPU= 23.509
  08:39:46 STEP 1192 H= 298:00 +CPU= 24.664
  08:40:10 STEP 1193 H= 298:15 +CPU= 23.866
  08:40:34 STEP 1194 H= 298:30 +CPU= 23.803
  08:40:58 STEP 1195 H= 298:45 +CPU= 24.357
  08:41:23 STEP 1196 H= 299:00 +CPU= 24.519
  08:41:47 STEP 1197 H= 299:15 +CPU= 24.261
  08:42:11 STEP 1198 H= 299:30 +CPU= 24.423
  08:42:36 STEP 1199 H= 299:45 +CPU= 24.370
  08:43:10 STEP 1200 H= 300:00 +CPU= 34.364
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001200
  08:43:34 STEP 1201 H= 300:15 +CPU= 24.101
  08:43:59 STEP 1202 H= 300:30 +CPU= 24.240
  08:44:23 STEP 1203 H= 300:45 +CPU= 24.452
  08:44:48 STEP 1204 H= 301:00 +CPU= 24.840
  08:45:12 STEP 1205 H= 301:15 +CPU= 24.162
  08:45:36 STEP 1206 H= 301:30 +CPU= 24.402
  08:46:01 STEP 1207 H= 301:45 +CPU= 24.371
  08:46:26 STEP 1208 H= 302:00 +CPU= 24.834
  08:46:50 STEP 1209 H= 302:15 +CPU= 24.214
  08:47:14 STEP 1210 H= 302:30 +CPU= 24.312
  08:47:39 STEP 1211 H= 302:45 +CPU= 24.758
  08:48:09 STEP 1212 H= 303:00 +CPU= 30.174
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001212
  08:48:33 STEP 1213 H= 303:15 +CPU= 23.764
  08:48:57 STEP 1214 H= 303:30 +CPU= 24.296
  08:49:22 STEP 1215 H= 303:45 +CPU= 24.291
  08:49:47 STEP 1216 H= 304:00 +CPU= 25.408
  08:50:11 STEP 1217 H= 304:15 +CPU= 23.833
  08:50:35 STEP 1218 H= 304:30 +CPU= 24.256
  08:51:00 STEP 1219 H= 304:45 +CPU= 24.840
  08:51:25 STEP 1220 H= 305:00 +CPU= 24.968
  08:51:49 STEP 1221 H= 305:15 +CPU= 24.503
  08:52:14 STEP 1222 H= 305:30 +CPU= 24.722
  08:52:38 STEP 1223 H= 305:45 +CPU= 24.141
  08:53:08 STEP 1224 H= 306:00 +CPU= 30.164
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001224
  08:53:33 STEP 1225 H= 306:15 +CPU= 24.388
  08:53:58 STEP 1226 H= 306:30 +CPU= 24.764
  08:54:22 STEP 1227 H= 306:45 +CPU= 24.368
  08:54:47 STEP 1228 H= 307:00 +CPU= 24.678
  08:55:11 STEP 1229 H= 307:15 +CPU= 24.359
  08:55:35 STEP 1230 H= 307:30 +CPU= 24.357
  08:56:00 STEP 1231 H= 307:45 +CPU= 24.631
  08:56:24 STEP 1232 H= 308:00 +CPU= 24.339
  08:56:49 STEP 1233 H= 308:15 +CPU= 24.384
  08:57:13 STEP 1234 H= 308:30 +CPU= 24.154
  08:57:37 STEP 1235 H= 308:45 +CPU= 23.797
  08:58:06 STEP 1236 H= 309:00 +CPU= 28.741
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001236
  08:58:30 STEP 1237 H= 309:15 +CPU= 24.495
  08:58:54 STEP 1238 H= 309:30 +CPU= 24.318
  08:59:19 STEP 1239 H= 309:45 +CPU= 24.961
  08:59:44 STEP 1240 H= 310:00 +CPU= 24.663
  09:00:09 STEP 1241 H= 310:15 +CPU= 24.761
  09:00:33 STEP 1242 H= 310:30 +CPU= 24.099
  09:00:57 STEP 1243 H= 310:45 +CPU= 24.347
  09:01:21 STEP 1244 H= 311:00 +CPU= 24.204
  09:01:46 STEP 1245 H= 311:15 +CPU= 24.179
  09:02:10 STEP 1246 H= 311:30 +CPU= 24.253
  09:02:34 STEP 1247 H= 311:45 +CPU= 23.875
  09:03:09 STEP 1248 H= 312:00 +CPU= 35.391
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0_r894036332_12.zip
Uploading the intermediate file: upload_file_12.zip
  09:03:34 STEP 1249 H= 312:15 +CPU= 24.869
  09:03:59 STEP 1250 H= 312:30 +CPU= 25.367
  09:04:24 STEP 1251 H= 312:45 +CPU= 24.789
  09:04:49 STEP 1252 H= 313:00 +CPU= 24.748
  09:05:13 STEP 1253 H= 313:15 +CPU= 23.776
  09:05:37 STEP 1254 H= 313:30 +CPU= 24.348
  09:06:02 STEP 1255 H= 313:45 +CPU= 24.658
  09:06:26 STEP 1256 H= 314:00 +CPU= 24.570
  09:06:50 STEP 1257 H= 314:15 +CPU= 23.941
  09:07:15 STEP 1258 H= 314:30 +CPU= 24.837
  09:07:39 STEP 1259 H= 314:45 +CPU= 23.666
  09:08:08 STEP 1260 H= 315:00 +CPU= 29.090
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001260
  09:08:32 STEP 1261 H= 315:15 +CPU= 23.873
  09:08:56 STEP 1262 H= 315:30 +CPU= 24.059
  09:09:21 STEP 1263 H= 315:45 +CPU= 24.770
  09:09:45 STEP 1264 H= 316:00 +CPU= 24.483
  09:10:09 STEP 1265 H= 316:15 +CPU= 23.615
  09:10:32 STEP 1266 H= 316:30 +CPU= 23.716
  09:10:57 STEP 1267 H= 316:45 +CPU= 24.712
  09:11:23 STEP 1268 H= 317:00 +CPU= 25.395
  09:11:47 STEP 1269 H= 317:15 +CPU= 23.949
  09:12:11 STEP 1270 H= 317:30 +CPU= 24.655
  09:12:35 STEP 1271 H= 317:45 +CPU= 24.248
  09:13:06 STEP 1272 H= 318:00 +CPU= 30.524
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001272
  09:13:30 STEP 1273 H= 318:15 +CPU= 24.236
  09:13:55 STEP 1274 H= 318:30 +CPU= 24.330
  09:14:19 STEP 1275 H= 318:45 +CPU= 23.983
  09:14:43 STEP 1276 H= 319:00 +CPU= 24.607
  09:15:08 STEP 1277 H= 319:15 +CPU= 24.506
  09:15:32 STEP 1278 H= 319:30 +CPU= 24.050
  09:15:56 STEP 1279 H= 319:45 +CPU= 24.216
  09:16:22 STEP 1280 H= 320:00 +CPU= 25.587
  09:16:46 STEP 1281 H= 320:15 +CPU= 24.090
  09:17:10 STEP 1282 H= 320:30 +CPU= 24.156
  09:17:34 STEP 1283 H= 320:45 +CPU= 23.986
  09:18:04 STEP 1284 H= 321:00 +CPU= 29.821
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001284
  09:18:28 STEP 1285 H= 321:15 +CPU= 24.036
  09:18:53 STEP 1286 H= 321:30 +CPU= 24.818
  09:19:17 STEP 1287 H= 321:45 +CPU= 24.493
  09:19:42 STEP 1288 H= 322:00 +CPU= 25.186
  09:20:07 STEP 1289 H= 322:15 +CPU= 24.308
  09:20:31 STEP 1290 H= 322:30 +CPU= 24.149
  09:20:55 STEP 1291 H= 322:45 +CPU= 24.560
  09:21:20 STEP 1292 H= 323:00 +CPU= 24.981
  09:21:45 STEP 1293 H= 323:15 +CPU= 24.773
  09:22:10 STEP 1294 H= 323:30 +CPU= 24.766
  09:22:34 STEP 1295 H= 323:45 +CPU= 24.375
  09:23:10 STEP 1296 H= 324:00 +CPU= 35.784
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001296
  09:23:34 STEP 1297 H= 324:15 +CPU= 24.503
  09:23:58 STEP 1298 H= 324:30 +CPU= 23.927
  09:24:23 STEP 1299 H= 324:45 +CPU= 24.774
  09:24:48 STEP 1300 H= 325:00 +CPU= 25.059
  09:25:12 STEP 1301 H= 325:15 +CPU= 23.701
  09:25:36 STEP 1302 H= 325:30 +CPU= 23.979
  09:26:00 STEP 1303 H= 325:45 +CPU= 24.109
  09:26:25 STEP 1304 H= 326:00 +CPU= 24.672
  09:26:49 STEP 1305 H= 326:15 +CPU= 24.071
  09:27:13 STEP 1306 H= 326:30 +CPU= 23.870
  09:27:36 STEP 1307 H= 326:45 +CPU= 23.735
  09:28:06 STEP 1308 H= 327:00 +CPU= 29.677
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001308
  09:28:30 STEP 1309 H= 327:15 +CPU= 23.849
  09:28:54 STEP 1310 H= 327:30 +CPU= 23.568
  09:29:17 STEP 1311 H= 327:45 +CPU= 23.540
  09:29:42 STEP 1312 H= 328:00 +CPU= 24.482
  09:30:06 STEP 1313 H= 328:15 +CPU= 24.003
  09:30:29 STEP 1314 H= 328:30 +CPU= 23.730
  09:30:54 STEP 1315 H= 328:45 +CPU= 24.328
  09:31:18 STEP 1316 H= 329:00 +CPU= 24.517
  09:31:43 STEP 1317 H= 329:15 +CPU= 24.565
  09:32:08 STEP 1318 H= 329:30 +CPU= 25.107
  09:32:33 STEP 1319 H= 329:45 +CPU= 24.678
  09:33:02 STEP 1320 H= 330:00 +CPU= 29.825
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001320
  09:33:26 STEP 1321 H= 330:15 +CPU= 23.924
  09:33:50 STEP 1322 H= 330:30 +CPU= 23.902
  09:34:15 STEP 1323 H= 330:45 +CPU= 24.472
  09:34:39 STEP 1324 H= 331:00 +CPU= 24.406
  09:35:03 STEP 1325 H= 331:15 +CPU= 24.068
  09:35:28 STEP 1326 H= 331:30 +CPU= 24.631
  09:35:53 STEP 1327 H= 331:45 +CPU= 25.149
  09:36:18 STEP 1328 H= 332:00 +CPU= 25.173
  09:36:43 STEP 1329 H= 332:15 +CPU= 25.121
  09:37:09 STEP 1330 H= 332:30 +CPU= 25.303
  09:37:33 STEP 1331 H= 332:45 +CPU= 24.779
  09:38:02 STEP 1332 H= 333:00 +CPU= 28.715
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001332
  09:38:26 STEP 1333 H= 333:15 +CPU= 23.915
  09:38:50 STEP 1334 H= 333:30 +CPU= 24.153
  09:39:15 STEP 1335 H= 333:45 +CPU= 24.717
  09:39:40 STEP 1336 H= 334:00 +CPU= 24.859
  09:40:04 STEP 1337 H= 334:15 +CPU= 24.471
  09:40:29 STEP 1338 H= 334:30 +CPU= 24.343
  09:40:53 STEP 1339 H= 334:45 +CPU= 24.805
  09:41:19 STEP 1340 H= 335:00 +CPU= 25.676
  09:41:43 STEP 1341 H= 335:15 +CPU= 24.254
  09:42:08 STEP 1342 H= 335:30 +CPU= 24.575
  09:42:33 STEP 1343 H= 335:45 +CPU= 24.877
  09:43:08 STEP 1344 H= 336:00 +CPU= 34.718
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0_r894036332_13.zip
Uploading the intermediate file: upload_file_13.zip
  09:43:32 STEP 1345 H= 336:15 +CPU= 24.251
  09:43:56 STEP 1346 H= 336:30 +CPU= 24.298
  09:44:21 STEP 1347 H= 336:45 +CPU= 24.409
  09:44:46 STEP 1348 H= 337:00 +CPU= 25.105
  09:45:11 STEP 1349 H= 337:15 +CPU= 24.869
  09:45:35 STEP 1350 H= 337:30 +CPU= 24.076
  09:45:59 STEP 1351 H= 337:45 +CPU= 23.975
  09:46:23 STEP 1352 H= 338:00 +CPU= 24.895
  09:46:48 STEP 1353 H= 338:15 +CPU= 24.554
  09:47:13 STEP 1354 H= 338:30 +CPU= 24.881
  09:47:37 STEP 1355 H= 338:45 +CPU= 24.091
  09:48:06 STEP 1356 H= 339:00 +CPU= 28.779
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001356
  09:48:30 STEP 1357 H= 339:15 +CPU= 24.218
  09:48:54 STEP 1358 H= 339:30 +CPU= 24.277
  09:49:18 STEP 1359 H= 339:45 +CPU= 24.183
  09:49:43 STEP 1360 H= 340:00 +CPU= 24.847
  09:50:08 STEP 1361 H= 340:15 +CPU= 24.359
  09:50:32 STEP 1362 H= 340:30 +CPU= 24.296
  09:50:56 STEP 1363 H= 340:45 +CPU= 23.594
  09:51:21 STEP 1364 H= 341:00 +CPU= 25.110
  09:51:45 STEP 1365 H= 341:15 +CPU= 24.210
  09:52:10 STEP 1366 H= 341:30 +CPU= 24.640
  09:52:34 STEP 1367 H= 341:45 +CPU= 24.580
  09:53:04 STEP 1368 H= 342:00 +CPU= 29.447
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001368
  09:53:29 STEP 1369 H= 342:15 +CPU= 24.849
  09:53:53 STEP 1370 H= 342:30 +CPU= 24.195
  09:54:17 STEP 1371 H= 342:45 +CPU= 24.309
  09:54:42 STEP 1372 H= 343:00 +CPU= 24.671
  09:55:07 STEP 1373 H= 343:15 +CPU= 25.504
  09:55:32 STEP 1374 H= 343:30 +CPU= 24.978
  09:55:56 STEP 1375 H= 343:45 +CPU= 24.116
  09:56:21 STEP 1376 H= 344:00 +CPU= 24.585
  09:56:45 STEP 1377 H= 344:15 +CPU= 24.566
  09:57:10 STEP 1378 H= 344:30 +CPU= 24.827
  09:57:34 STEP 1379 H= 344:45 +CPU= 23.925
  09:58:03 STEP 1380 H= 345:00 +CPU= 28.425
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001380
  09:58:28 STEP 1381 H= 345:15 +CPU= 24.876
  09:58:52 STEP 1382 H= 345:30 +CPU= 24.689
  09:59:17 STEP 1383 H= 345:45 +CPU= 24.505
  09:59:41 STEP 1384 H= 346:00 +CPU= 24.503
  10:00:06 STEP 1385 H= 346:15 +CPU= 24.525
  10:00:30 STEP 1386 H= 346:30 +CPU= 24.613
  10:00:54 STEP 1387 H= 346:45 +CPU= 23.566
  10:01:19 STEP 1388 H= 347:00 +CPU= 25.224
  10:01:44 STEP 1389 H= 347:15 +CPU= 24.817
  10:02:09 STEP 1390 H= 347:30 +CPU= 25.389
  10:02:34 STEP 1391 H= 347:45 +CPU= 24.223
  10:03:08 STEP 1392 H= 348:00 +CPU= 34.227
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001392
  10:03:32 STEP 1393 H= 348:15 +CPU= 24.343
  10:03:56 STEP 1394 H= 348:30 +CPU= 23.973
  10:04:21 STEP 1395 H= 348:45 +CPU= 24.453
  10:04:46 STEP 1396 H= 349:00 +CPU= 25.001
  10:05:10 STEP 1397 H= 349:15 +CPU= 24.500
  10:05:35 STEP 1398 H= 349:30 +CPU= 24.875
  10:06:00 STEP 1399 H= 349:45 +CPU= 25.035
  10:06:26 STEP 1400 H= 350:00 +CPU= 26.008
  10:06:51 STEP 1401 H= 350:15 +CPU= 24.845
  10:07:16 STEP 1402 H= 350:30 +CPU= 24.523
  10:07:40 STEP 1403 H= 350:45 +CPU= 24.112
  10:08:09 STEP 1404 H= 351:00 +CPU= 29.679
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001404
  10:08:34 STEP 1405 H= 351:15 +CPU= 24.444
  10:08:58 STEP 1406 H= 351:30 +CPU= 24.207
  10:09:22 STEP 1407 H= 351:45 +CPU= 23.665
  10:09:46 STEP 1408 H= 352:00 +CPU= 23.848
  10:10:09 STEP 1409 H= 352:15 +CPU= 23.518
  10:10:33 STEP 1410 H= 352:30 +CPU= 24.341
  10:10:57 STEP 1411 H= 352:45 +CPU= 24.028
  10:11:22 STEP 1412 H= 353:00 +CPU= 24.911
  10:11:47 STEP 1413 H= 353:15 +CPU= 24.371
  10:12:11 STEP 1414 H= 353:30 +CPU= 24.221
  10:12:35 STEP 1415 H= 353:45 +CPU= 24.139
  10:13:05 STEP 1416 H= 354:00 +CPU= 29.429
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001416
  10:13:29 STEP 1417 H= 354:15 +CPU= 24.931
  10:13:54 STEP 1418 H= 354:30 +CPU= 24.051
  10:14:17 STEP 1419 H= 354:45 +CPU= 23.953
  10:14:42 STEP 1420 H= 355:00 +CPU= 24.827
  10:15:07 STEP 1421 H= 355:15 +CPU= 24.439
  10:15:31 STEP 1422 H= 355:30 +CPU= 24.083
  10:15:55 STEP 1423 H= 355:45 +CPU= 24.054
  10:16:20 STEP 1424 H= 356:00 +CPU= 25.245
  10:16:45 STEP 1425 H= 356:15 +CPU= 24.365
  10:17:09 STEP 1426 H= 356:30 +CPU= 24.650
  10:17:34 STEP 1427 H= 356:45 +CPU= 24.949
  10:18:03 STEP 1428 H= 357:00 +CPU= 28.978
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001428
  10:18:28 STEP 1429 H= 357:15 +CPU= 24.569
  10:18:52 STEP 1430 H= 357:30 +CPU= 24.744
  10:19:18 STEP 1431 H= 357:45 +CPU= 25.492
  10:19:43 STEP 1432 H= 358:00 +CPU= 25.006
  10:20:08 STEP 1433 H= 358:15 +CPU= 24.598
  10:20:33 STEP 1434 H= 358:30 +CPU= 25.229
  10:20:58 STEP 1435 H= 358:45 +CPU= 25.212
  10:21:23 STEP 1436 H= 359:00 +CPU= 25.267
  10:21:48 STEP 1437 H= 359:15 +CPU= 24.364
  10:22:13 STEP 1438 H= 359:30 +CPU= 24.899
  10:22:38 STEP 1439 H= 359:45 +CPU= 25.032
  10:23:08 STEP 1440 H= 360:00 +CPU= 30.210
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1383071145E-20      I       0.8477435444E-07      I
ICloud fractionI      0.0000000000E+00      I       0.5490385729E-02      I
I  Relat. Hum. I      0.7368922793E-01      I       0.9742369914E-01      I
I  PASS 01     I      0.0000000000E+00      I       0.1535002480E+02      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    305348708 BYTES IN      7 RECORDS TRANSFERRED IN    0.0011 SECONDS277589.7 Mbytes/s, TOTAL TIME=    0.0111( 9.9%)
 10:23:08 STEP 1440 H= 360:00 +CPU= 30.210
 END CNT3
 NSTEP =  1440 CNT0   000000000  21.082  21.082  21.085   1.000               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        11440/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    203    0       46      1044068    11440 ************    24.8     1.3    23.6     1.8    42.4
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 8585994.5 8585994.5    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 8583364.2 8583364.2     99.97      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  417    4165.6    4165.6     20.23      0.00
   9 SPCM     - SPECTRAL COMP.                     384    1124.9    1124.9      5.03      0.00
  10 SCAN2M - PHYSICS                              385    9045.7    9045.7     40.56      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 32    1129.5    1129.5      0.42      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 17      61.2      61.2      0.01      0.00
  17 GRIDFPOS IN CNT4                               32      28.8      28.8      0.01      0.00
  27 RESTART FILES - WRITING                         8    4517.8    4517.8      0.42      0.00
  28 RESTART FILES - READING                         1    2187.2    2187.2      0.03      0.00
  29 SU4FPOS IN CNT4                                32       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                32    4197.7    4197.7      1.56      0.00
  31 POSDDH IN STEPO                                 2       5.3       5.3      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    385    4623.4    4623.4     20.73      0.00
  39 SU0YOMB                                         1     272.3     272.3      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    385      27.8      27.8      0.12      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              384     612.0     612.0      2.74      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              384      69.9      69.9      0.31      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 17       0.6       0.6      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      836     237.5     237.5      2.31      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       800     141.7     141.7      1.32      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        800      76.1      76.1      0.71      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       836     100.1     100.1      0.97      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      21.9      21.9      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            836      92.6      92.6      0.90      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            800      55.6      55.6      0.52      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            836      87.0      87.0      0.85      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            800      47.5      47.5      0.44      0.00
 400 GSTATS                                      45508       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 42288       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     8586.0 CPU TIME    8583.0 VECTOR TIME     8583.0


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      151.0
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/3/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/3/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284229/ICMGGh7zg+001440
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0_r894036332_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
10:26:35 (5467): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0_r894036332_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0_r894036332_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0_r894036332_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0_r894036332_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0_r894036332_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Jun 2024 16:23:01 1523278 22435809 oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0 1,295,100 37,070 0.0286
09 Jun 2024 16:23:01 1523278 22435809 oifs_43r3_bl_a1uq_2016092300_20_1017_12284229_0 864,000 25,596 0.0296


©2024 climateprediction.net