climateprediction.net home page
Task 22435855

Task 22435855

Name oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0
Workunit 12284272
Created 7 Jun 2024, 14:22:53 UTC
Sent 7 Jun 2024, 14:30:13 UTC
Report deadline 6 Aug 2024, 14:30:13 UTC
Received 9 Jun 2024, 16:21:50 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1523278
Run time 10 hours 22 min 10 sec
CPU time 10 hours 22 min 10 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 8.64 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,013.09 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,645.08 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
5 +CPU= 22.685
  14:06:06 STEP 1022 H= 255:30 +CPU= 24.755
  14:06:32 STEP 1023 H= 255:45 +CPU= 25.578
  14:06:57 STEP 1024 H= 256:00 +CPU= 24.744
  14:07:21 STEP 1025 H= 256:15 +CPU= 23.632
  14:07:45 STEP 1026 H= 256:30 +CPU= 24.452
  14:08:09 STEP 1027 H= 256:45 +CPU= 24.689
  14:08:35 STEP 1028 H= 257:00 +CPU= 25.143
  14:09:00 STEP 1029 H= 257:15 +CPU= 24.846
  14:09:25 STEP 1030 H= 257:30 +CPU= 24.732
  14:09:50 STEP 1031 H= 257:45 +CPU= 25.009
  14:10:21 STEP 1032 H= 258:00 +CPU= 30.552
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001032
  14:10:46 STEP 1033 H= 258:15 +CPU= 25.519
  14:11:10 STEP 1034 H= 258:30 +CPU= 24.818
  14:11:35 STEP 1035 H= 258:45 +CPU= 25.056
  14:12:01 STEP 1036 H= 259:00 +CPU= 25.152
  14:12:27 STEP 1037 H= 259:15 +CPU= 26.682
  14:12:54 STEP 1038 H= 259:30 +CPU= 26.315
  14:13:20 STEP 1039 H= 259:45 +CPU= 25.653
  14:13:45 STEP 1040 H= 260:00 +CPU= 25.879
  14:14:10 STEP 1041 H= 260:15 +CPU= 24.605
  14:14:34 STEP 1042 H= 260:30 +CPU= 24.913
  14:14:59 STEP 1043 H= 260:45 +CPU= 24.251
  14:15:30 STEP 1044 H= 261:00 +CPU= 30.720
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001044
  14:15:55 STEP 1045 H= 261:15 +CPU= 24.949
  14:16:21 STEP 1046 H= 261:30 +CPU= 25.863
  14:16:49 STEP 1047 H= 261:45 +CPU= 27.964
  14:17:20 STEP 1048 H= 262:00 +CPU= 29.621
  14:17:46 STEP 1049 H= 262:15 +CPU= 27.450
  14:18:14 STEP 1050 H= 262:30 +CPU= 27.301
  14:18:40 STEP 1051 H= 262:45 +CPU= 26.555
  14:19:04 STEP 1052 H= 263:00 +CPU= 25.296
  14:19:30 STEP 1053 H= 263:15 +CPU= 25.348
  14:19:57 STEP 1054 H= 263:30 +CPU= 26.685
  14:20:22 STEP 1055 H= 263:45 +CPU= 25.569
  14:21:00 STEP 1056 H= 264:00 +CPU= 36.091
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0_r2044002603_10.zip
Uploading the intermediate file: upload_file_10.zip
  14:21:25 STEP 1057 H= 264:15 +CPU= 25.692
  14:21:50 STEP 1058 H= 264:30 +CPU= 24.732
  14:22:15 STEP 1059 H= 264:45 +CPU= 25.884
  14:22:43 STEP 1060 H= 265:00 +CPU= 28.619
  14:23:12 STEP 1061 H= 265:15 +CPU= 27.787
  14:23:40 STEP 1062 H= 265:30 +CPU= 28.298
  14:24:08 STEP 1063 H= 265:45 +CPU= 28.363
  14:24:37 STEP 1064 H= 266:00 +CPU= 28.195
  14:25:05 STEP 1065 H= 266:15 +CPU= 28.761
  14:25:32 STEP 1066 H= 266:30 +CPU= 26.850
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.847] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 5751MB (heap), 5023MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.847] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.847] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.847] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.848] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.848] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 5751 MB (maxheap), 5023 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     1        (1004) 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               LAPINEA 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142541:1717849541:78911.849] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                LARMES 
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142542:1717849542:78912.849] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94104:94104] [20240608:142542:1717849542:78912.849] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D5E56B  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000D870B3  larmes_                   455  larmes.F90
oifs_43r3_model.e  0000000000D5968D  lapinea_                  270  lapinea.F90
oifs_43r3_model.e  0000000000D56B78  call_sl_                  300  call_sl.F90
oifs_43r3_model.e  0000000000D54BBC  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BD4FD8  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1vx
(argv4) batchid: 1017
(argv5) wuid: 12284272
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_65a461f2e95e703be886024dacacf576 to: /var/lib/boinc-client/slots/5/oifs_43r3_bl_a1vx_2016092300_20_1017_12284272.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_bl_a1vx_2016092300_20_1017_12284272.zip
ic_ancil_file: ic_ancil_12284272
ifsdata_file: ifsdata_12284272
climate_data_file: clim_data_12284272
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/5/ic_ancil_12284272.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12284272.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_84e43de5146609d20b18ba18a72b96f7 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12284272.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12284272.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12284272.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12284272.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12284272.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12284272.xml
-- Model is restarting --
Adjusting last_iter, 1067, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12284272.xml
last_cpu_time: 28486
upload_file_number: 11
last_iter: 1057
last_upload: 950400
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0_r2044002603
The child process has been launched with process id: 5590
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5590:5590] [20240609:080253:1717912973:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:03:07 STEP 1056 H= 264:00 +CPU= 11.620
  08:03:18 STEP 1057 H= 264:15 +CPU= 10.896
  08:03:29 STEP 1058 H= 264:30 +CPU= 10.841
  08:03:40 STEP 1059 H= 264:45 +CPU= 10.901
  08:03:51 STEP 1060 H= 265:00 +CPU= 11.344
  08:04:02 STEP 1061 H= 265:15 +CPU= 10.833
  08:04:13 STEP 1062 H= 265:30 +CPU= 10.900
  08:04:24 STEP 1063 H= 265:45 +CPU= 11.020
  08:04:35 STEP 1064 H= 266:00 +CPU= 11.335
  08:04:46 STEP 1065 H= 266:15 +CPU= 11.049
  08:04:58 STEP 1066 H= 266:30 +CPU= 11.222
  08:05:09 STEP 1067 H= 266:45 +CPU= 11.265
  08:05:24 STEP 1068 H= 267:00 +CPU= 15.081
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001068
  08:05:35 STEP 1069 H= 267:15 +CPU= 11.413
  08:05:47 STEP 1070 H= 267:30 +CPU= 11.504
  08:05:59 STEP 1071 H= 267:45 +CPU= 11.580
  08:06:10 STEP 1072 H= 268:00 +CPU= 11.702
  08:06:22 STEP 1073 H= 268:15 +CPU= 11.549
  08:06:33 STEP 1074 H= 268:30 +CPU= 11.604
  08:06:45 STEP 1075 H= 268:45 +CPU= 11.455
  08:06:57 STEP 1076 H= 269:00 +CPU= 11.976
  08:07:08 STEP 1077 H= 269:15 +CPU= 11.472
  08:07:20 STEP 1078 H= 269:30 +CPU= 11.670
  08:07:32 STEP 1079 H= 269:45 +CPU= 11.594
  08:07:47 STEP 1080 H= 270:00 +CPU= 15.121
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001080
  08:07:59 STEP 1081 H= 270:15 +CPU= 11.818
  08:08:10 STEP 1082 H= 270:30 +CPU= 11.654
  08:08:22 STEP 1083 H= 270:45 +CPU= 11.607
  08:08:34 STEP 1084 H= 271:00 +CPU= 11.989
  08:08:45 STEP 1085 H= 271:15 +CPU= 11.585
  08:08:57 STEP 1086 H= 271:30 +CPU= 11.476
  08:09:08 STEP 1087 H= 271:45 +CPU= 11.623
  08:09:20 STEP 1088 H= 272:00 +CPU= 11.926
  08:09:32 STEP 1089 H= 272:15 +CPU= 11.543
  08:09:43 STEP 1090 H= 272:30 +CPU= 11.627
  08:09:55 STEP 1091 H= 272:45 +CPU= 11.714
  08:10:11 STEP 1092 H= 273:00 +CPU= 15.792
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001092
  08:10:23 STEP 1093 H= 273:15 +CPU= 12.148
  08:10:35 STEP 1094 H= 273:30 +CPU= 12.024
  08:10:47 STEP 1095 H= 273:45 +CPU= 12.149
  08:11:00 STEP 1096 H= 274:00 +CPU= 12.477
  08:11:12 STEP 1097 H= 274:15 +CPU= 12.465
  08:11:25 STEP 1098 H= 274:30 +CPU= 12.783
  08:11:38 STEP 1099 H= 274:45 +CPU= 12.594
  08:11:51 STEP 1100 H= 275:00 +CPU= 13.258
  08:12:04 STEP 1101 H= 275:15 +CPU= 12.778
  08:12:16 STEP 1102 H= 275:30 +CPU= 12.846
  08:12:29 STEP 1103 H= 275:45 +CPU= 12.594
  08:12:48 STEP 1104 H= 276:00 +CPU= 18.980
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001104
  08:13:01 STEP 1105 H= 276:15 +CPU= 12.787
  08:13:13 STEP 1106 H= 276:30 +CPU= 12.482
  08:13:26 STEP 1107 H= 276:45 +CPU= 12.348
  08:13:39 STEP 1108 H= 277:00 +CPU= 12.629
  08:13:51 STEP 1109 H= 277:15 +CPU= 12.668
  08:14:05 STEP 1110 H= 277:30 +CPU= 12.804
  08:14:18 STEP 1111 H= 277:45 +CPU= 13.096
  08:14:32 STEP 1112 H= 278:00 +CPU= 13.889
  08:14:46 STEP 1113 H= 278:15 +CPU= 13.963
  08:15:00 STEP 1114 H= 278:30 +CPU= 14.483
  08:15:15 STEP 1115 H= 278:45 +CPU= 14.697
  08:15:34 STEP 1116 H= 279:00 +CPU= 19.441
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001116
  08:15:51 STEP 1117 H= 279:15 +CPU= 15.958
  08:16:07 STEP 1118 H= 279:30 +CPU= 16.036
  08:16:22 STEP 1119 H= 279:45 +CPU= 16.372
  08:16:41 STEP 1120 H= 280:00 +CPU= 18.498
  08:16:58 STEP 1121 H= 280:15 +CPU= 17.627
  08:17:18 STEP 1122 H= 280:30 +CPU= 19.836
  08:17:40 STEP 1123 H= 280:45 +CPU= 20.834
  08:18:01 STEP 1124 H= 281:00 +CPU= 21.992
  08:18:23 STEP 1125 H= 281:15 +CPU= 22.386
  08:18:48 STEP 1126 H= 281:30 +CPU= 24.778
  08:19:15 STEP 1127 H= 281:45 +CPU= 26.620
  08:19:47 STEP 1128 H= 282:00 +CPU= 31.457
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001128
  08:20:14 STEP 1129 H= 282:15 +CPU= 26.776
  08:20:39 STEP 1130 H= 282:30 +CPU= 25.749
  08:21:04 STEP 1131 H= 282:45 +CPU= 24.075
  08:21:29 STEP 1132 H= 283:00 +CPU= 25.883
  08:21:55 STEP 1133 H= 283:15 +CPU= 25.825
  08:22:21 STEP 1134 H= 283:30 +CPU= 26.430
  08:22:45 STEP 1135 H= 283:45 +CPU= 24.336
  08:23:11 STEP 1136 H= 284:00 +CPU= 25.316
  08:23:35 STEP 1137 H= 284:15 +CPU= 24.725
  08:24:02 STEP 1138 H= 284:30 +CPU= 26.863
  08:24:27 STEP 1139 H= 284:45 +CPU= 25.001
  08:24:58 STEP 1140 H= 285:00 +CPU= 30.642
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001140
  08:25:23 STEP 1141 H= 285:15 +CPU= 24.810
  08:25:50 STEP 1142 H= 285:30 +CPU= 26.987
  08:26:16 STEP 1143 H= 285:45 +CPU= 26.081
  08:26:42 STEP 1144 H= 286:00 +CPU= 25.917
  08:27:07 STEP 1145 H= 286:15 +CPU= 24.698
  08:27:31 STEP 1146 H= 286:30 +CPU= 24.299
  08:27:55 STEP 1147 H= 286:45 +CPU= 24.122
  08:28:19 STEP 1148 H= 287:00 +CPU= 24.107
  08:28:43 STEP 1149 H= 287:15 +CPU= 23.741
  08:29:07 STEP 1150 H= 287:30 +CPU= 24.481
  08:29:32 STEP 1151 H= 287:45 +CPU= 24.586
  08:30:07 STEP 1152 H= 288:00 +CPU= 35.107
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0_r2044002603_11.zip
Uploading the intermediate file: upload_file_11.zip
  08:30:32 STEP 1153 H= 288:15 +CPU= 24.620
  08:30:57 STEP 1154 H= 288:30 +CPU= 25.165
  08:31:21 STEP 1155 H= 288:45 +CPU= 24.516
  08:31:46 STEP 1156 H= 289:00 +CPU= 25.033
  08:32:11 STEP 1157 H= 289:15 +CPU= 24.221
  08:32:35 STEP 1158 H= 289:30 +CPU= 24.693
  08:33:00 STEP 1159 H= 289:45 +CPU= 24.704
  08:33:26 STEP 1160 H= 290:00 +CPU= 25.645
  08:33:51 STEP 1161 H= 290:15 +CPU= 24.950
  08:34:15 STEP 1162 H= 290:30 +CPU= 24.635
  08:34:39 STEP 1163 H= 290:45 +CPU= 24.066
  08:35:10 STEP 1164 H= 291:00 +CPU= 30.988
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001164
  08:35:36 STEP 1165 H= 291:15 +CPU= 25.225
  08:36:01 STEP 1166 H= 291:30 +CPU= 25.171
  08:36:25 STEP 1167 H= 291:45 +CPU= 24.299
  08:36:50 STEP 1168 H= 292:00 +CPU= 25.332
  08:37:14 STEP 1169 H= 292:15 +CPU= 23.898
  08:37:39 STEP 1170 H= 292:30 +CPU= 24.412
  08:38:05 STEP 1171 H= 292:45 +CPU= 26.314
  08:38:31 STEP 1172 H= 293:00 +CPU= 25.852
  08:38:57 STEP 1173 H= 293:15 +CPU= 25.656
  08:39:20 STEP 1174 H= 293:30 +CPU= 23.533
  08:39:47 STEP 1175 H= 293:45 +CPU= 26.520
  08:40:17 STEP 1176 H= 294:00 +CPU= 30.401
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001176
  08:40:42 STEP 1177 H= 294:15 +CPU= 24.952
  08:41:07 STEP 1178 H= 294:30 +CPU= 25.102
  08:41:32 STEP 1179 H= 294:45 +CPU= 24.773
  08:41:57 STEP 1180 H= 295:00 +CPU= 25.239
  08:42:22 STEP 1181 H= 295:15 +CPU= 24.794
  08:42:47 STEP 1182 H= 295:30 +CPU= 24.546
  08:43:11 STEP 1183 H= 295:45 +CPU= 24.551
  08:43:36 STEP 1184 H= 296:00 +CPU= 24.284
  08:44:00 STEP 1185 H= 296:15 +CPU= 24.480
  08:44:25 STEP 1186 H= 296:30 +CPU= 24.593
  08:44:49 STEP 1187 H= 296:45 +CPU= 24.423
  08:45:19 STEP 1188 H= 297:00 +CPU= 29.528
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001188
  08:45:43 STEP 1189 H= 297:15 +CPU= 24.712
  08:46:07 STEP 1190 H= 297:30 +CPU= 24.138
  08:46:32 STEP 1191 H= 297:45 +CPU= 24.667
  08:46:57 STEP 1192 H= 298:00 +CPU= 24.591
  08:47:21 STEP 1193 H= 298:15 +CPU= 24.541
  08:47:46 STEP 1194 H= 298:30 +CPU= 25.173
  08:48:11 STEP 1195 H= 298:45 +CPU= 24.541
  08:48:36 STEP 1196 H= 299:00 +CPU= 24.872
  08:49:00 STEP 1197 H= 299:15 +CPU= 24.091
  08:49:25 STEP 1198 H= 299:30 +CPU= 24.669
  08:49:48 STEP 1199 H= 299:45 +CPU= 23.799
  08:50:24 STEP 1200 H= 300:00 +CPU= 35.154
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001200
  08:50:48 STEP 1201 H= 300:15 +CPU= 24.678
  08:51:13 STEP 1202 H= 300:30 +CPU= 24.934
  08:51:38 STEP 1203 H= 300:45 +CPU= 24.556
  08:52:03 STEP 1204 H= 301:00 +CPU= 25.094
  08:52:27 STEP 1205 H= 301:15 +CPU= 24.411
  08:52:52 STEP 1206 H= 301:30 +CPU= 24.562
  08:53:16 STEP 1207 H= 301:45 +CPU= 24.529
  08:53:41 STEP 1208 H= 302:00 +CPU= 24.673
  08:54:06 STEP 1209 H= 302:15 +CPU= 24.922
  08:54:31 STEP 1210 H= 302:30 +CPU= 24.614
  08:54:56 STEP 1211 H= 302:45 +CPU= 24.948
  08:55:25 STEP 1212 H= 303:00 +CPU= 29.495
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001212
  08:55:49 STEP 1213 H= 303:15 +CPU= 24.305
  08:56:14 STEP 1214 H= 303:30 +CPU= 24.495
  08:56:39 STEP 1215 H= 303:45 +CPU= 24.583
  08:57:03 STEP 1216 H= 304:00 +CPU= 24.808
  08:57:29 STEP 1217 H= 304:15 +CPU= 25.280
  08:57:53 STEP 1218 H= 304:30 +CPU= 24.427
  08:58:18 STEP 1219 H= 304:45 +CPU= 24.451
  08:58:42 STEP 1220 H= 305:00 +CPU= 24.784
  08:59:08 STEP 1221 H= 305:15 +CPU= 25.377
  08:59:33 STEP 1222 H= 305:30 +CPU= 24.757
  08:59:58 STEP 1223 H= 305:45 +CPU= 24.987
  09:00:26 STEP 1224 H= 306:00 +CPU= 28.765
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001224
  09:00:51 STEP 1225 H= 306:15 +CPU= 24.624
  09:01:15 STEP 1226 H= 306:30 +CPU= 24.441
  09:01:39 STEP 1227 H= 306:45 +CPU= 24.143
  09:02:05 STEP 1228 H= 307:00 +CPU= 25.058
  09:02:29 STEP 1229 H= 307:15 +CPU= 24.360
  09:02:54 STEP 1230 H= 307:30 +CPU= 25.019
  09:03:19 STEP 1231 H= 307:45 +CPU= 25.008
  09:03:44 STEP 1232 H= 308:00 +CPU= 24.785
  09:04:09 STEP 1233 H= 308:15 +CPU= 25.296
  09:04:34 STEP 1234 H= 308:30 +CPU= 25.269
  09:04:59 STEP 1235 H= 308:45 +CPU= 24.906
  09:05:29 STEP 1236 H= 309:00 +CPU= 29.808
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001236
  09:05:54 STEP 1237 H= 309:15 +CPU= 25.356
  09:06:19 STEP 1238 H= 309:30 +CPU= 24.575
  09:06:43 STEP 1239 H= 309:45 +CPU= 24.200
  09:07:08 STEP 1240 H= 310:00 +CPU= 24.844
  09:07:33 STEP 1241 H= 310:15 +CPU= 24.697
  09:07:57 STEP 1242 H= 310:30 +CPU= 24.525
  09:08:22 STEP 1243 H= 310:45 +CPU= 24.491
  09:08:46 STEP 1244 H= 311:00 +CPU= 24.128
  09:09:11 STEP 1245 H= 311:15 +CPU= 24.904
  09:09:36 STEP 1246 H= 311:30 +CPU= 25.203
  09:10:01 STEP 1247 H= 311:45 +CPU= 24.603
  09:10:36 STEP 1248 H= 312:00 +CPU= 35.296
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0_r2044002603_12.zip
Uploading the intermediate file: upload_file_12.zip
  09:11:01 STEP 1249 H= 312:15 +CPU= 25.501
  09:11:26 STEP 1250 H= 312:30 +CPU= 24.579
  09:11:50 STEP 1251 H= 312:45 +CPU= 24.375
  09:12:15 STEP 1252 H= 313:00 +CPU= 24.874
  09:12:40 STEP 1253 H= 313:15 +CPU= 24.620
  09:13:04 STEP 1254 H= 313:30 +CPU= 24.512
  09:13:29 STEP 1255 H= 313:45 +CPU= 24.114
  09:13:53 STEP 1256 H= 314:00 +CPU= 24.443
  09:14:18 STEP 1257 H= 314:15 +CPU= 24.777
  09:14:42 STEP 1258 H= 314:30 +CPU= 23.918
  09:15:07 STEP 1259 H= 314:45 +CPU= 25.002
  09:15:36 STEP 1260 H= 315:00 +CPU= 29.289
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001260
  09:16:00 STEP 1261 H= 315:15 +CPU= 23.827
  09:16:24 STEP 1262 H= 315:30 +CPU= 24.332
  09:16:49 STEP 1263 H= 315:45 +CPU= 24.559
  09:17:13 STEP 1264 H= 316:00 +CPU= 24.316
  09:17:37 STEP 1265 H= 316:15 +CPU= 24.450
  09:18:02 STEP 1266 H= 316:30 +CPU= 24.572
  09:18:26 STEP 1267 H= 316:45 +CPU= 24.288
  09:18:50 STEP 1268 H= 317:00 +CPU= 23.827
  09:19:15 STEP 1269 H= 317:15 +CPU= 24.723
  09:19:40 STEP 1270 H= 317:30 +CPU= 24.602
  09:20:05 STEP 1271 H= 317:45 +CPU= 25.658
  09:20:36 STEP 1272 H= 318:00 +CPU= 30.635
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001272
  09:21:00 STEP 1273 H= 318:15 +CPU= 24.398
  09:21:25 STEP 1274 H= 318:30 +CPU= 24.618
  09:21:50 STEP 1275 H= 318:45 +CPU= 24.709
  09:22:14 STEP 1276 H= 319:00 +CPU= 24.539
  09:22:39 STEP 1277 H= 319:15 +CPU= 24.668
  09:23:03 STEP 1278 H= 319:30 +CPU= 24.390
  09:23:27 STEP 1279 H= 319:45 +CPU= 24.111
  09:23:52 STEP 1280 H= 320:00 +CPU= 24.549
  09:24:17 STEP 1281 H= 320:15 +CPU= 25.024
  09:24:41 STEP 1282 H= 320:30 +CPU= 24.371
  09:25:06 STEP 1283 H= 320:45 +CPU= 24.396
  09:25:36 STEP 1284 H= 321:00 +CPU= 29.935
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001284
  09:26:00 STEP 1285 H= 321:15 +CPU= 24.295
  09:26:25 STEP 1286 H= 321:30 +CPU= 24.630
  09:26:49 STEP 1287 H= 321:45 +CPU= 24.585
  09:27:14 STEP 1288 H= 322:00 +CPU= 24.494
  09:27:38 STEP 1289 H= 322:15 +CPU= 24.656
  09:28:02 STEP 1290 H= 322:30 +CPU= 23.876
  09:28:27 STEP 1291 H= 322:45 +CPU= 24.374
  09:28:52 STEP 1292 H= 323:00 +CPU= 24.969
  09:29:16 STEP 1293 H= 323:15 +CPU= 23.975
  09:29:40 STEP 1294 H= 323:30 +CPU= 24.482
  09:30:04 STEP 1295 H= 323:45 +CPU= 24.034
  09:30:40 STEP 1296 H= 324:00 +CPU= 35.440
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001296
  09:31:04 STEP 1297 H= 324:15 +CPU= 24.336
  09:31:28 STEP 1298 H= 324:30 +CPU= 24.466
  09:31:53 STEP 1299 H= 324:45 +CPU= 24.369
  09:32:18 STEP 1300 H= 325:00 +CPU= 25.077
  09:32:43 STEP 1301 H= 325:15 +CPU= 24.918
  09:33:07 STEP 1302 H= 325:30 +CPU= 24.405
  09:33:32 STEP 1303 H= 325:45 +CPU= 24.639
  09:33:57 STEP 1304 H= 326:00 +CPU= 24.777
  09:34:21 STEP 1305 H= 326:15 +CPU= 24.111
  09:34:45 STEP 1306 H= 326:30 +CPU= 24.399
  09:35:09 STEP 1307 H= 326:45 +CPU= 24.266
  09:35:39 STEP 1308 H= 327:00 +CPU= 29.730
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001308
  09:36:04 STEP 1309 H= 327:15 +CPU= 24.688
  09:36:29 STEP 1310 H= 327:30 +CPU= 25.020
  09:36:53 STEP 1311 H= 327:45 +CPU= 24.489
  09:37:17 STEP 1312 H= 328:00 +CPU= 24.094
  09:37:43 STEP 1313 H= 328:15 +CPU= 25.130
  09:38:07 STEP 1314 H= 328:30 +CPU= 24.973
  09:38:32 STEP 1315 H= 328:45 +CPU= 24.985
  09:38:57 STEP 1316 H= 329:00 +CPU= 24.517
  09:39:21 STEP 1317 H= 329:15 +CPU= 24.042
  09:39:46 STEP 1318 H= 329:30 +CPU= 24.477
  09:40:12 STEP 1319 H= 329:45 +CPU= 26.073
  09:40:41 STEP 1320 H= 330:00 +CPU= 29.716
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001320
  09:41:06 STEP 1321 H= 330:15 +CPU= 24.426
  09:41:31 STEP 1322 H= 330:30 +CPU= 25.240
  09:41:56 STEP 1323 H= 330:45 +CPU= 25.364
  09:42:21 STEP 1324 H= 331:00 +CPU= 24.986
  09:42:46 STEP 1325 H= 331:15 +CPU= 24.974
  09:43:11 STEP 1326 H= 331:30 +CPU= 24.630
  09:43:36 STEP 1327 H= 331:45 +CPU= 24.964
  09:44:01 STEP 1328 H= 332:00 +CPU= 25.049
  09:44:26 STEP 1329 H= 332:15 +CPU= 24.465
  09:44:50 STEP 1330 H= 332:30 +CPU= 24.119
  09:45:14 STEP 1331 H= 332:45 +CPU= 24.268
  09:45:43 STEP 1332 H= 333:00 +CPU= 29.423
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001332
  09:46:08 STEP 1333 H= 333:15 +CPU= 24.752
  09:46:33 STEP 1334 H= 333:30 +CPU= 24.730
  09:46:58 STEP 1335 H= 333:45 +CPU= 24.963
  09:47:23 STEP 1336 H= 334:00 +CPU= 24.948
  09:47:48 STEP 1337 H= 334:15 +CPU= 24.828
  09:48:12 STEP 1338 H= 334:30 +CPU= 24.370
  09:48:37 STEP 1339 H= 334:45 +CPU= 24.546
  09:49:01 STEP 1340 H= 335:00 +CPU= 24.776
  09:49:25 STEP 1341 H= 335:15 +CPU= 23.368
  09:49:50 STEP 1342 H= 335:30 +CPU= 24.848
  09:50:14 STEP 1343 H= 335:45 +CPU= 24.306
  09:50:49 STEP 1344 H= 336:00 +CPU= 35.124
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0_r2044002603_13.zip
Uploading the intermediate file: upload_file_13.zip
  09:51:13 STEP 1345 H= 336:15 +CPU= 24.366
  09:51:37 STEP 1346 H= 336:30 +CPU= 23.908
  09:52:01 STEP 1347 H= 336:45 +CPU= 24.230
  09:52:26 STEP 1348 H= 337:00 +CPU= 24.869
  09:52:51 STEP 1349 H= 337:15 +CPU= 24.262
  09:53:16 STEP 1350 H= 337:30 +CPU= 25.082
  09:53:41 STEP 1351 H= 337:45 +CPU= 25.104
  09:54:06 STEP 1352 H= 338:00 +CPU= 25.365
  09:54:31 STEP 1353 H= 338:15 +CPU= 25.019
  09:54:57 STEP 1354 H= 338:30 +CPU= 25.439
  09:55:22 STEP 1355 H= 338:45 +CPU= 25.111
  09:55:52 STEP 1356 H= 339:00 +CPU= 30.198
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001356
  09:56:16 STEP 1357 H= 339:15 +CPU= 24.184
  09:56:41 STEP 1358 H= 339:30 +CPU= 25.056
  09:57:06 STEP 1359 H= 339:45 +CPU= 24.668
  09:57:31 STEP 1360 H= 340:00 +CPU= 25.163
  09:57:56 STEP 1361 H= 340:15 +CPU= 24.473
  09:58:20 STEP 1362 H= 340:30 +CPU= 24.568
  09:58:45 STEP 1363 H= 340:45 +CPU= 24.868
  09:59:10 STEP 1364 H= 341:00 +CPU= 24.844
  09:59:34 STEP 1365 H= 341:15 +CPU= 24.149
  09:59:58 STEP 1366 H= 341:30 +CPU= 24.243
  10:00:23 STEP 1367 H= 341:45 +CPU= 25.075
  10:00:54 STEP 1368 H= 342:00 +CPU= 30.617
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001368
  10:01:19 STEP 1369 H= 342:15 +CPU= 24.669
  10:01:43 STEP 1370 H= 342:30 +CPU= 24.568
  10:02:08 STEP 1371 H= 342:45 +CPU= 24.671
  10:02:33 STEP 1372 H= 343:00 +CPU= 25.091
  10:02:57 STEP 1373 H= 343:15 +CPU= 23.804
  10:03:22 STEP 1374 H= 343:30 +CPU= 24.899
  10:03:46 STEP 1375 H= 343:45 +CPU= 24.629
  10:04:11 STEP 1376 H= 344:00 +CPU= 24.971
  10:04:36 STEP 1377 H= 344:15 +CPU= 24.597
  10:05:00 STEP 1378 H= 344:30 +CPU= 24.259
  10:05:25 STEP 1379 H= 344:45 +CPU= 25.250
  10:05:55 STEP 1380 H= 345:00 +CPU= 29.382
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001380
  10:06:20 STEP 1381 H= 345:15 +CPU= 25.627
  10:06:45 STEP 1382 H= 345:30 +CPU= 24.948
  10:07:11 STEP 1383 H= 345:45 +CPU= 25.153
  10:07:35 STEP 1384 H= 346:00 +CPU= 24.514
  10:08:00 STEP 1385 H= 346:15 +CPU= 24.851
  10:08:24 STEP 1386 H= 346:30 +CPU= 24.224
  10:08:49 STEP 1387 H= 346:45 +CPU= 24.477
  10:09:14 STEP 1388 H= 347:00 +CPU= 24.901
  10:09:38 STEP 1389 H= 347:15 +CPU= 24.263
  10:10:02 STEP 1390 H= 347:30 +CPU= 24.469
  10:10:27 STEP 1391 H= 347:45 +CPU= 24.258
  10:11:01 STEP 1392 H= 348:00 +CPU= 34.839
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001392
  10:11:26 STEP 1393 H= 348:15 +CPU= 24.688
  10:11:51 STEP 1394 H= 348:30 +CPU= 24.895
  10:12:16 STEP 1395 H= 348:45 +CPU= 24.772
  10:12:40 STEP 1396 H= 349:00 +CPU= 24.659
  10:13:05 STEP 1397 H= 349:15 +CPU= 24.578
  10:13:30 STEP 1398 H= 349:30 +CPU= 25.082
  10:13:54 STEP 1399 H= 349:45 +CPU= 24.282
  10:14:20 STEP 1400 H= 350:00 +CPU= 25.442
  10:14:44 STEP 1401 H= 350:15 +CPU= 24.098
  10:15:09 STEP 1402 H= 350:30 +CPU= 24.945
  10:15:34 STEP 1403 H= 350:45 +CPU= 24.565
  10:16:03 STEP 1404 H= 351:00 +CPU= 29.094
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001404
  10:16:28 STEP 1405 H= 351:15 +CPU= 25.506
  10:16:53 STEP 1406 H= 351:30 +CPU= 24.608
  10:17:18 STEP 1407 H= 351:45 +CPU= 25.105
  10:17:42 STEP 1408 H= 352:00 +CPU= 24.566
  10:18:08 STEP 1409 H= 352:15 +CPU= 25.428
  10:18:33 STEP 1410 H= 352:30 +CPU= 24.660
  10:18:58 STEP 1411 H= 352:45 +CPU= 25.066
  10:19:22 STEP 1412 H= 353:00 +CPU= 24.723
  10:19:47 STEP 1413 H= 353:15 +CPU= 24.136
  10:20:11 STEP 1414 H= 353:30 +CPU= 24.926
  10:20:37 STEP 1415 H= 353:45 +CPU= 25.124
  10:21:07 STEP 1416 H= 354:00 +CPU= 30.578
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001416
  10:21:32 STEP 1417 H= 354:15 +CPU= 25.093
  10:21:57 STEP 1418 H= 354:30 +CPU= 24.973
  10:22:22 STEP 1419 H= 354:45 +CPU= 24.558
  10:22:47 STEP 1420 H= 355:00 +CPU= 25.421
  10:23:11 STEP 1421 H= 355:15 +CPU= 23.727
  10:23:33 STEP 1422 H= 355:30 +CPU= 22.208
  10:23:55 STEP 1423 H= 355:45 +CPU= 22.070
  10:24:18 STEP 1424 H= 356:00 +CPU= 22.342
  10:24:40 STEP 1425 H= 356:15 +CPU= 21.906
  10:25:02 STEP 1426 H= 356:30 +CPU= 22.442
  10:25:22 STEP 1427 H= 356:45 +CPU= 19.678
  10:25:45 STEP 1428 H= 357:00 +CPU= 23.781
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001428
  10:26:05 STEP 1429 H= 357:15 +CPU= 19.668
  10:26:25 STEP 1430 H= 357:30 +CPU= 19.596
  10:26:45 STEP 1431 H= 357:45 +CPU= 19.947
  10:27:07 STEP 1432 H= 358:00 +CPU= 22.100
  10:27:28 STEP 1433 H= 358:15 +CPU= 21.695
  10:27:50 STEP 1434 H= 358:30 +CPU= 21.729
  10:28:12 STEP 1435 H= 358:45 +CPU= 21.987
  10:28:34 STEP 1436 H= 359:00 +CPU= 22.296
  10:28:59 STEP 1437 H= 359:15 +CPU= 24.052
  10:29:24 STEP 1438 H= 359:30 +CPU= 25.116
  10:29:49 STEP 1439 H= 359:45 +CPU= 24.870
  10:30:20 STEP 1440 H= 360:00 +CPU= 31.676
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.4414803738E-21      I       0.2405060619E-05      I
ICloud fractionI      0.0000000000E+00      I       0.4360530784E-01      I
I  Relat. Hum. I      0.4818174846E+00      I       0.5190613828E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.3874855451E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    309629028 BYTES IN      7 RECORDS TRANSFERRED IN    0.0008 SECONDS387036.3 Mbytes/s, TOTAL TIME=    0.0094( 8.5%)
 10:30:20 STEP 1440 H= 360:00 +CPU= 31.676
 END CNT3
 NSTEP =  1440 CNT0   000000000  22.434  22.434  22.460   0.999               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        11440/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    203    0       46      1044068    11440 ************    25.6     1.2    24.3     1.7    45.0
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 8847093.6 8847093.6    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 8844579.5 8844579.5     99.97      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  417    4209.1    4209.1     19.84      0.00
   9 SPCM     - SPECTRAL COMP.                     384    1201.1    1201.1      5.21      0.00
  10 SCAN2M - PHYSICS                              385    9500.3    9500.3     41.34      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 32    1155.4    1155.4      0.42      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 17      62.9      62.9      0.01      0.00
  17 GRIDFPOS IN CNT4                               32      28.9      28.9      0.01      0.00
  27 RESTART FILES - WRITING                         8    4621.4    4621.4      0.42      0.00
  28 RESTART FILES - READING                         1    2088.5    2088.5      0.02      0.00
  29 SU4FPOS IN CNT4                                32       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                32    4331.0    4331.0      1.57      0.00
  31 POSDDH IN STEPO                                 2       2.2       2.2      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    385    4658.0    4658.0     20.27      0.00
  39 SU0YOMB                                         1     273.3     273.3      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    385      28.1      28.1      0.12      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              384     587.2     587.2      2.55      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              384      73.5      73.5      0.32      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 17       0.5       0.5      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      836     239.3     239.3      2.26      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       800     145.2     145.2      1.31      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        800      80.6      80.6      0.73      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       836     104.6     104.6      0.99      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      22.4      22.4      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            836      96.6      96.6      0.91      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            800      59.7      59.7      0.54      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            836      91.9      91.9      0.87      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            800      51.4      51.4      0.47      0.00
 400 GSTATS                                      45508       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 42288       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     8847.1 CPU TIME    8844.3 VECTOR TIME     8844.3


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      146.5
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/5/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/5/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284272/ICMGGh7zg+001440
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0_r2044002603_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
10:33:47 (5587): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0_r2044002603_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0_r2044002603_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0_r2044002603_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0_r2044002603_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0_r2044002603_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Jun 2024 16:23:00 1523278 22435855 oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0 1,295,100 37,330 0.0288
09 Jun 2024 16:23:00 1523278 22435855 oifs_43r3_bl_a1vx_2016092300_20_1017_12284272_0 864,000 25,645 0.0297


©2024 cpdn.org