climateprediction.net home page
Task 22435772

Task 22435772

Name oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0
Workunit 12284192
Created 7 Jun 2024, 14:16:01 UTC
Sent 7 Jun 2024, 14:30:13 UTC
Report deadline 6 Aug 2024, 14:30:13 UTC
Received 9 Jun 2024, 16:21:50 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1523278
Run time 10 hours 19 min 35 sec
CPU time 10 hours 19 min 35 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 8.64 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,019.90 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,674.86 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
1038 H= 259:30 +CPU= 24.993
  14:12:21 STEP 1039 H= 259:45 +CPU= 25.927
  14:12:47 STEP 1040 H= 260:00 +CPU= 26.389
  14:13:12 STEP 1041 H= 260:15 +CPU= 24.599
  14:13:36 STEP 1042 H= 260:30 +CPU= 25.437
  14:14:02 STEP 1043 H= 260:45 +CPU= 24.559
  14:14:32 STEP 1044 H= 261:00 +CPU= 30.524
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001044
  14:14:57 STEP 1045 H= 261:15 +CPU= 25.191
  14:15:22 STEP 1046 H= 261:30 +CPU= 24.309
  14:15:47 STEP 1047 H= 261:45 +CPU= 25.276
  14:16:12 STEP 1048 H= 262:00 +CPU= 25.805
  14:16:40 STEP 1049 H= 262:15 +CPU= 26.977
  14:17:09 STEP 1050 H= 262:30 +CPU= 27.748
  14:17:36 STEP 1051 H= 262:45 +CPU= 27.819
  14:18:04 STEP 1052 H= 263:00 +CPU= 28.268
  14:18:31 STEP 1053 H= 263:15 +CPU= 26.970
  14:18:56 STEP 1054 H= 263:30 +CPU= 25.085
  14:19:20 STEP 1055 H= 263:45 +CPU= 25.034
  14:19:56 STEP 1056 H= 264:00 +CPU= 35.928
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0_r1726112566_10.zip
Uploading the intermediate file: upload_file_10.zip
  14:20:21 STEP 1057 H= 264:15 +CPU= 25.031
  14:20:48 STEP 1058 H= 264:30 +CPU= 25.594
  14:21:14 STEP 1059 H= 264:45 +CPU= 26.056
  14:21:40 STEP 1060 H= 265:00 +CPU= 25.592
  14:22:05 STEP 1061 H= 265:15 +CPU= 25.893
  14:22:33 STEP 1062 H= 265:30 +CPU= 28.713
  14:23:02 STEP 1063 H= 265:45 +CPU= 28.576
  14:23:31 STEP 1064 H= 266:00 +CPU= 28.839
  14:23:59 STEP 1065 H= 266:15 +CPU= 28.011
  14:24:28 STEP 1066 H= 266:30 +CPU= 28.118
  14:24:55 STEP 1067 H= 266:45 +CPU= 27.297
  14:25:28 STEP 1068 H= 267:00 +CPU= 33.221
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001068
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.567] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 5751MB (heap), 5022MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.568] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.568] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.568] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.569] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.569] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 5751 MB (maxheap), 5022 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.570] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     2        (1005) 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               LAPINEB 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                LARCINB 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 LAITRE_GMV 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142541:1717849541:78899.571] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  LAITRI 
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142542:1717849542:78900.571] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94112:94112] [20240608:142542:1717849542:78900.571] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000DCACFC  laitri_                   343  laitri.F90
oifs_43r3_model.e  0000000000DC6EE4  laitre_gmv_               143  laitre_gmv.F90
oifs_43r3_model.e  0000000000DCE6AB  larcinb_                  301  larcinb.F90
oifs_43r3_model.e  0000000000D8A8F8  lapineb_                  451  lapineb.F90
oifs_43r3_model.e  0000000000D57FC1  call_sl_                  363  call_sl.F90
oifs_43r3_model.e  0000000000D54BBC  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BD4FD8  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1tp
(argv4) batchid: 1017
(argv5) wuid: 12284192
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/10
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_95c3925238bd29c980a48bc2de3510ac to: /var/lib/boinc-client/slots/10/oifs_43r3_bl_a1tp_2016092300_20_1017_12284192.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_bl_a1tp_2016092300_20_1017_12284192.zip
ic_ancil_file: ic_ancil_12284192
ifsdata_file: ifsdata_12284192
climate_data_file: clim_data_12284192
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/10/ic_ancil_12284192.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12284192.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4329c200529d15f89028b8ff911a5fa7 to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12284192.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12284192.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12284192.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12284192.zip
Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12284192.xml
Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12284192.xml
-- Model is restarting --
Adjusting last_iter, 1069, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12284192.xml
last_cpu_time: 28477
upload_file_number: 11
last_iter: 1057
last_upload: 950400
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0_r1726112566
The child process has been launched with process id: 5483
Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5483:5483] [20240609:080002:1717912802:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:00:16 STEP 1056 H= 264:00 +CPU= 11.598
  08:00:27 STEP 1057 H= 264:15 +CPU= 10.813
  08:00:38 STEP 1058 H= 264:30 +CPU= 10.807
  08:00:48 STEP 1059 H= 264:45 +CPU= 10.801
  08:01:00 STEP 1060 H= 265:00 +CPU= 11.171
  08:01:10 STEP 1061 H= 265:15 +CPU= 10.793
  08:01:21 STEP 1062 H= 265:30 +CPU= 10.783
  08:01:32 STEP 1063 H= 265:45 +CPU= 10.793
  08:01:43 STEP 1064 H= 266:00 +CPU= 11.126
  08:01:54 STEP 1065 H= 266:15 +CPU= 10.822
  08:02:05 STEP 1066 H= 266:30 +CPU= 10.800
  08:02:16 STEP 1067 H= 266:45 +CPU= 10.764
  08:02:30 STEP 1068 H= 267:00 +CPU= 14.291
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001068
  08:02:41 STEP 1069 H= 267:15 +CPU= 10.850
  08:02:52 STEP 1070 H= 267:30 +CPU= 10.881
  08:03:03 STEP 1071 H= 267:45 +CPU= 11.002
  08:03:14 STEP 1072 H= 268:00 +CPU= 11.377
  08:03:25 STEP 1073 H= 268:15 +CPU= 10.996
  08:03:36 STEP 1074 H= 268:30 +CPU= 11.086
  08:03:47 STEP 1075 H= 268:45 +CPU= 11.061
  08:03:59 STEP 1076 H= 269:00 +CPU= 11.458
  08:04:10 STEP 1077 H= 269:15 +CPU= 11.105
  08:04:21 STEP 1078 H= 269:30 +CPU= 11.219
  08:04:32 STEP 1079 H= 269:45 +CPU= 11.306
  08:04:48 STEP 1080 H= 270:00 +CPU= 15.329
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001080
  08:04:59 STEP 1081 H= 270:15 +CPU= 11.488
  08:05:11 STEP 1082 H= 270:30 +CPU= 11.383
  08:05:22 STEP 1083 H= 270:45 +CPU= 11.494
  08:05:34 STEP 1084 H= 271:00 +CPU= 11.937
  08:05:45 STEP 1085 H= 271:15 +CPU= 11.436
  08:05:57 STEP 1086 H= 271:30 +CPU= 11.408
  08:06:08 STEP 1087 H= 271:45 +CPU= 11.462
  08:06:20 STEP 1088 H= 272:00 +CPU= 12.003
  08:06:32 STEP 1089 H= 272:15 +CPU= 11.684
  08:06:44 STEP 1090 H= 272:30 +CPU= 11.642
  08:06:55 STEP 1091 H= 272:45 +CPU= 11.370
  08:07:10 STEP 1092 H= 273:00 +CPU= 15.392
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001092
  08:07:22 STEP 1093 H= 273:15 +CPU= 11.602
  08:07:34 STEP 1094 H= 273:30 +CPU= 11.733
  08:07:45 STEP 1095 H= 273:45 +CPU= 11.518
  08:07:57 STEP 1096 H= 274:00 +CPU= 11.887
  08:08:09 STEP 1097 H= 274:15 +CPU= 11.637
  08:08:21 STEP 1098 H= 274:30 +CPU= 11.673
  08:08:32 STEP 1099 H= 274:45 +CPU= 11.653
  08:08:44 STEP 1100 H= 275:00 +CPU= 11.869
  08:08:55 STEP 1101 H= 275:15 +CPU= 11.557
  08:09:07 STEP 1102 H= 275:30 +CPU= 11.625
  08:09:19 STEP 1103 H= 275:45 +CPU= 11.691
  08:09:36 STEP 1104 H= 276:00 +CPU= 17.749
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001104
  08:09:48 STEP 1105 H= 276:15 +CPU= 11.713
  08:10:00 STEP 1106 H= 276:30 +CPU= 11.742
  08:10:12 STEP 1107 H= 276:45 +CPU= 11.783
  08:10:24 STEP 1108 H= 277:00 +CPU= 12.586
  08:10:36 STEP 1109 H= 277:15 +CPU= 12.081
  08:10:48 STEP 1110 H= 277:30 +CPU= 11.981
  08:11:00 STEP 1111 H= 277:45 +CPU= 12.024
  08:11:13 STEP 1112 H= 278:00 +CPU= 12.552
  08:11:26 STEP 1113 H= 278:15 +CPU= 12.759
  08:11:38 STEP 1114 H= 278:30 +CPU= 12.630
  08:11:51 STEP 1115 H= 278:45 +CPU= 12.821
  08:12:07 STEP 1116 H= 279:00 +CPU= 16.264
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001116
  08:12:20 STEP 1117 H= 279:15 +CPU= 12.922
  08:12:33 STEP 1118 H= 279:30 +CPU= 12.671
  08:12:46 STEP 1119 H= 279:45 +CPU= 12.520
  08:12:59 STEP 1120 H= 280:00 +CPU= 13.106
  08:13:11 STEP 1121 H= 280:15 +CPU= 12.467
  08:13:23 STEP 1122 H= 280:30 +CPU= 12.251
  08:13:36 STEP 1123 H= 280:45 +CPU= 12.611
  08:13:49 STEP 1124 H= 281:00 +CPU= 13.082
  08:14:03 STEP 1125 H= 281:15 +CPU= 12.865
  08:14:16 STEP 1126 H= 281:30 +CPU= 13.521
  08:14:30 STEP 1127 H= 281:45 +CPU= 13.394
  08:14:48 STEP 1128 H= 282:00 +CPU= 18.319
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001128
  08:15:03 STEP 1129 H= 282:15 +CPU= 14.662
  08:15:17 STEP 1130 H= 282:30 +CPU= 14.296
  08:15:32 STEP 1131 H= 282:45 +CPU= 14.834
  08:15:49 STEP 1132 H= 283:00 +CPU= 16.704
  08:16:05 STEP 1133 H= 283:15 +CPU= 16.271
  08:16:21 STEP 1134 H= 283:30 +CPU= 16.689
  08:16:39 STEP 1135 H= 283:45 +CPU= 18.156
  08:16:58 STEP 1136 H= 284:00 +CPU= 19.186
  08:17:18 STEP 1137 H= 284:15 +CPU= 19.209
  08:17:38 STEP 1138 H= 284:30 +CPU= 20.113
  08:18:00 STEP 1139 H= 284:45 +CPU= 21.792
  08:18:26 STEP 1140 H= 285:00 +CPU= 26.632
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001140
  08:18:52 STEP 1141 H= 285:15 +CPU= 25.029
  08:19:17 STEP 1142 H= 285:30 +CPU= 25.770
  08:19:45 STEP 1143 H= 285:45 +CPU= 27.655
  08:20:12 STEP 1144 H= 286:00 +CPU= 26.816
  08:20:38 STEP 1145 H= 286:15 +CPU= 26.109
  08:21:03 STEP 1146 H= 286:30 +CPU= 24.717
  08:21:28 STEP 1147 H= 286:45 +CPU= 25.758
  08:21:54 STEP 1148 H= 287:00 +CPU= 26.121
  08:22:19 STEP 1149 H= 287:15 +CPU= 24.626
  08:22:43 STEP 1150 H= 287:30 +CPU= 24.674
  08:23:08 STEP 1151 H= 287:45 +CPU= 24.845
  08:23:43 STEP 1152 H= 288:00 +CPU= 34.998
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0_r1726112566_11.zip
Uploading the intermediate file: upload_file_11.zip
  08:24:10 STEP 1153 H= 288:15 +CPU= 26.632
  08:24:35 STEP 1154 H= 288:30 +CPU= 25.207
  08:25:01 STEP 1155 H= 288:45 +CPU= 26.180
  08:25:29 STEP 1156 H= 289:00 +CPU= 27.465
  08:25:55 STEP 1157 H= 289:15 +CPU= 25.888
  08:26:21 STEP 1158 H= 289:30 +CPU= 26.216
  08:26:46 STEP 1159 H= 289:45 +CPU= 25.230
  08:27:11 STEP 1160 H= 290:00 +CPU= 24.845
  08:27:36 STEP 1161 H= 290:15 +CPU= 24.868
  08:28:00 STEP 1162 H= 290:30 +CPU= 24.350
  08:28:25 STEP 1163 H= 290:45 +CPU= 24.531
  08:28:54 STEP 1164 H= 291:00 +CPU= 29.446
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001164
  08:29:19 STEP 1165 H= 291:15 +CPU= 24.769
  08:29:43 STEP 1166 H= 291:30 +CPU= 24.217
  08:30:08 STEP 1167 H= 291:45 +CPU= 24.496
  08:30:33 STEP 1168 H= 292:00 +CPU= 24.894
  08:30:57 STEP 1169 H= 292:15 +CPU= 24.483
  08:31:21 STEP 1170 H= 292:30 +CPU= 24.111
  08:31:46 STEP 1171 H= 292:45 +CPU= 24.813
  08:32:10 STEP 1172 H= 293:00 +CPU= 24.125
  08:32:35 STEP 1173 H= 293:15 +CPU= 24.652
  08:32:59 STEP 1174 H= 293:30 +CPU= 24.426
  08:33:24 STEP 1175 H= 293:45 +CPU= 24.286
  08:33:54 STEP 1176 H= 294:00 +CPU= 29.784
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001176
  08:34:18 STEP 1177 H= 294:15 +CPU= 24.779
  08:34:43 STEP 1178 H= 294:30 +CPU= 24.487
  08:35:07 STEP 1179 H= 294:45 +CPU= 24.049
  08:35:32 STEP 1180 H= 295:00 +CPU= 25.219
  08:35:57 STEP 1181 H= 295:15 +CPU= 24.657
  08:36:21 STEP 1182 H= 295:30 +CPU= 23.832
  08:36:45 STEP 1183 H= 295:45 +CPU= 24.653
  08:37:10 STEP 1184 H= 296:00 +CPU= 25.127
  08:37:34 STEP 1185 H= 296:15 +CPU= 23.940
  08:38:00 STEP 1186 H= 296:30 +CPU= 25.947
  08:38:27 STEP 1187 H= 296:45 +CPU= 26.621
  08:38:58 STEP 1188 H= 297:00 +CPU= 31.099
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001188
  08:39:24 STEP 1189 H= 297:15 +CPU= 25.611
  08:39:48 STEP 1190 H= 297:30 +CPU= 24.284
  08:40:13 STEP 1191 H= 297:45 +CPU= 25.420
  08:40:38 STEP 1192 H= 298:00 +CPU= 25.007
  08:41:03 STEP 1193 H= 298:15 +CPU= 24.427
  08:41:27 STEP 1194 H= 298:30 +CPU= 24.576
  08:41:52 STEP 1195 H= 298:45 +CPU= 24.490
  08:42:17 STEP 1196 H= 299:00 +CPU= 25.021
  08:42:41 STEP 1197 H= 299:15 +CPU= 24.377
  08:43:05 STEP 1198 H= 299:30 +CPU= 24.043
  08:43:30 STEP 1199 H= 299:45 +CPU= 24.229
  08:44:04 STEP 1200 H= 300:00 +CPU= 34.731
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001200
  08:44:29 STEP 1201 H= 300:15 +CPU= 24.362
  08:44:53 STEP 1202 H= 300:30 +CPU= 23.864
  08:45:16 STEP 1203 H= 300:45 +CPU= 23.786
  08:45:42 STEP 1204 H= 301:00 +CPU= 25.306
  08:46:06 STEP 1205 H= 301:15 +CPU= 24.624
  08:46:31 STEP 1206 H= 301:30 +CPU= 25.018
  08:46:56 STEP 1207 H= 301:45 +CPU= 24.606
  08:47:21 STEP 1208 H= 302:00 +CPU= 24.808
  08:47:46 STEP 1209 H= 302:15 +CPU= 24.867
  08:48:11 STEP 1210 H= 302:30 +CPU= 24.810
  08:48:35 STEP 1211 H= 302:45 +CPU= 24.878
  08:49:05 STEP 1212 H= 303:00 +CPU= 29.276
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001212
  08:49:29 STEP 1213 H= 303:15 +CPU= 24.210
  08:49:53 STEP 1214 H= 303:30 +CPU= 24.527
  08:50:18 STEP 1215 H= 303:45 +CPU= 24.172
  08:50:42 STEP 1216 H= 304:00 +CPU= 23.859
  08:51:06 STEP 1217 H= 304:15 +CPU= 24.724
  08:51:31 STEP 1218 H= 304:30 +CPU= 24.977
  08:51:56 STEP 1219 H= 304:45 +CPU= 25.001
  08:52:21 STEP 1220 H= 305:00 +CPU= 24.693
  08:52:45 STEP 1221 H= 305:15 +CPU= 24.355
  08:53:10 STEP 1222 H= 305:30 +CPU= 24.477
  08:53:34 STEP 1223 H= 305:45 +CPU= 24.415
  08:54:04 STEP 1224 H= 306:00 +CPU= 30.193
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001224
  08:54:29 STEP 1225 H= 306:15 +CPU= 24.166
  08:54:53 STEP 1226 H= 306:30 +CPU= 24.155
  08:55:16 STEP 1227 H= 306:45 +CPU= 23.485
  08:55:40 STEP 1228 H= 307:00 +CPU= 24.130
  08:56:05 STEP 1229 H= 307:15 +CPU= 24.788
  08:56:30 STEP 1230 H= 307:30 +CPU= 25.106
  08:56:54 STEP 1231 H= 307:45 +CPU= 23.807
  08:57:19 STEP 1232 H= 308:00 +CPU= 25.218
  08:57:44 STEP 1233 H= 308:15 +CPU= 24.881
  08:58:09 STEP 1234 H= 308:30 +CPU= 25.157
  08:58:34 STEP 1235 H= 308:45 +CPU= 24.503
  08:59:04 STEP 1236 H= 309:00 +CPU= 30.513
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001236
  08:59:29 STEP 1237 H= 309:15 +CPU= 24.947
  08:59:54 STEP 1238 H= 309:30 +CPU= 24.614
  09:00:17 STEP 1239 H= 309:45 +CPU= 23.528
  09:00:42 STEP 1240 H= 310:00 +CPU= 24.598
  09:01:07 STEP 1241 H= 310:15 +CPU= 24.707
  09:01:32 STEP 1242 H= 310:30 +CPU= 25.217
  09:01:56 STEP 1243 H= 310:45 +CPU= 24.309
  09:02:22 STEP 1244 H= 311:00 +CPU= 25.173
  09:02:46 STEP 1245 H= 311:15 +CPU= 24.485
  09:03:11 STEP 1246 H= 311:30 +CPU= 24.817
  09:03:35 STEP 1247 H= 311:45 +CPU= 24.325
  09:04:10 STEP 1248 H= 312:00 +CPU= 34.488
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0_r1726112566_12.zip
Uploading the intermediate file: upload_file_12.zip
  09:04:34 STEP 1249 H= 312:15 +CPU= 24.761
  09:04:59 STEP 1250 H= 312:30 +CPU= 24.499
  09:05:23 STEP 1251 H= 312:45 +CPU= 23.844
  09:05:48 STEP 1252 H= 313:00 +CPU= 25.171
  09:06:13 STEP 1253 H= 313:15 +CPU= 25.101
  09:06:38 STEP 1254 H= 313:30 +CPU= 24.580
  09:07:02 STEP 1255 H= 313:45 +CPU= 24.080
  09:07:27 STEP 1256 H= 314:00 +CPU= 24.893
  09:07:52 STEP 1257 H= 314:15 +CPU= 24.998
  09:08:16 STEP 1258 H= 314:30 +CPU= 24.729
  09:08:41 STEP 1259 H= 314:45 +CPU= 24.310
  09:09:11 STEP 1260 H= 315:00 +CPU= 29.884
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001260
  09:09:35 STEP 1261 H= 315:15 +CPU= 24.743
  09:10:00 STEP 1262 H= 315:30 +CPU= 24.380
  09:10:24 STEP 1263 H= 315:45 +CPU= 24.116
  09:10:49 STEP 1264 H= 316:00 +CPU= 25.094
  09:11:15 STEP 1265 H= 316:15 +CPU= 25.572
  09:11:40 STEP 1266 H= 316:30 +CPU= 25.455
  09:12:05 STEP 1267 H= 316:45 +CPU= 24.789
  09:12:30 STEP 1268 H= 317:00 +CPU= 25.142
  09:12:54 STEP 1269 H= 317:15 +CPU= 24.583
  09:13:19 STEP 1270 H= 317:30 +CPU= 24.090
  09:13:43 STEP 1271 H= 317:45 +CPU= 24.638
  09:14:13 STEP 1272 H= 318:00 +CPU= 30.226
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001272
  09:14:38 STEP 1273 H= 318:15 +CPU= 24.623
  09:15:03 STEP 1274 H= 318:30 +CPU= 25.171
  09:15:28 STEP 1275 H= 318:45 +CPU= 24.695
  09:15:53 STEP 1276 H= 319:00 +CPU= 24.815
  09:16:18 STEP 1277 H= 319:15 +CPU= 24.942
  09:16:43 STEP 1278 H= 319:30 +CPU= 25.286
  09:17:07 STEP 1279 H= 319:45 +CPU= 24.442
  09:17:32 STEP 1280 H= 320:00 +CPU= 24.992
  09:17:57 STEP 1281 H= 320:15 +CPU= 24.140
  09:18:21 STEP 1282 H= 320:30 +CPU= 24.147
  09:18:45 STEP 1283 H= 320:45 +CPU= 24.361
  09:19:15 STEP 1284 H= 321:00 +CPU= 30.030
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001284
  09:19:40 STEP 1285 H= 321:15 +CPU= 24.334
  09:20:04 STEP 1286 H= 321:30 +CPU= 24.818
  09:20:29 STEP 1287 H= 321:45 +CPU= 24.386
  09:20:54 STEP 1288 H= 322:00 +CPU= 25.193
  09:21:18 STEP 1289 H= 322:15 +CPU= 24.335
  09:21:43 STEP 1290 H= 322:30 +CPU= 24.656
  09:22:08 STEP 1291 H= 322:45 +CPU= 25.348
  09:22:34 STEP 1292 H= 323:00 +CPU= 25.298
  09:22:58 STEP 1293 H= 323:15 +CPU= 24.747
  09:23:23 STEP 1294 H= 323:30 +CPU= 24.885
  09:23:48 STEP 1295 H= 323:45 +CPU= 24.489
  09:24:23 STEP 1296 H= 324:00 +CPU= 35.218
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001296
  09:24:47 STEP 1297 H= 324:15 +CPU= 24.488
  09:25:12 STEP 1298 H= 324:30 +CPU= 24.041
  09:25:36 STEP 1299 H= 324:45 +CPU= 24.241
  09:26:01 STEP 1300 H= 325:00 +CPU= 25.095
  09:26:26 STEP 1301 H= 325:15 +CPU= 24.700
  09:26:50 STEP 1302 H= 325:30 +CPU= 24.552
  09:27:15 STEP 1303 H= 325:45 +CPU= 24.459
  09:27:39 STEP 1304 H= 326:00 +CPU= 24.675
  09:28:04 STEP 1305 H= 326:15 +CPU= 24.528
  09:28:29 STEP 1306 H= 326:30 +CPU= 24.737
  09:28:53 STEP 1307 H= 326:45 +CPU= 24.877
  09:29:24 STEP 1308 H= 327:00 +CPU= 30.286
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001308
  09:29:48 STEP 1309 H= 327:15 +CPU= 24.359
  09:30:12 STEP 1310 H= 327:30 +CPU= 24.213
  09:30:36 STEP 1311 H= 327:45 +CPU= 23.520
  09:31:01 STEP 1312 H= 328:00 +CPU= 25.086
  09:31:26 STEP 1313 H= 328:15 +CPU= 24.933
  09:31:51 STEP 1314 H= 328:30 +CPU= 24.713
  09:32:15 STEP 1315 H= 328:45 +CPU= 24.276
  09:32:40 STEP 1316 H= 329:00 +CPU= 25.602
  09:33:05 STEP 1317 H= 329:15 +CPU= 24.875
  09:33:29 STEP 1318 H= 329:30 +CPU= 24.107
  09:33:54 STEP 1319 H= 329:45 +CPU= 24.513
  09:34:24 STEP 1320 H= 330:00 +CPU= 30.345
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001320
  09:34:49 STEP 1321 H= 330:15 +CPU= 24.814
  09:35:14 STEP 1322 H= 330:30 +CPU= 24.695
  09:35:38 STEP 1323 H= 330:45 +CPU= 23.968
  09:36:03 STEP 1324 H= 331:00 +CPU= 25.124
  09:36:28 STEP 1325 H= 331:15 +CPU= 25.122
  09:36:53 STEP 1326 H= 331:30 +CPU= 24.473
  09:37:17 STEP 1327 H= 331:45 +CPU= 24.212
  09:37:42 STEP 1328 H= 332:00 +CPU= 25.694
  09:38:07 STEP 1329 H= 332:15 +CPU= 24.893
  09:38:33 STEP 1330 H= 332:30 +CPU= 25.375
  09:38:58 STEP 1331 H= 332:45 +CPU= 25.423
  09:39:28 STEP 1332 H= 333:00 +CPU= 30.001
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001332
  09:39:53 STEP 1333 H= 333:15 +CPU= 24.288
  09:40:16 STEP 1334 H= 333:30 +CPU= 23.164
  09:40:40 STEP 1335 H= 333:45 +CPU= 24.256
  09:41:05 STEP 1336 H= 334:00 +CPU= 25.547
  09:41:31 STEP 1337 H= 334:15 +CPU= 25.668
  09:41:56 STEP 1338 H= 334:30 +CPU= 25.184
  09:42:21 STEP 1339 H= 334:45 +CPU= 24.700
  09:42:47 STEP 1340 H= 335:00 +CPU= 25.458
  09:43:12 STEP 1341 H= 335:15 +CPU= 25.415
  09:43:37 STEP 1342 H= 335:30 +CPU= 25.446
  09:44:02 STEP 1343 H= 335:45 +CPU= 25.036
  09:44:37 STEP 1344 H= 336:00 +CPU= 34.315
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0_r1726112566_13.zip
Uploading the intermediate file: upload_file_13.zip
  09:45:02 STEP 1345 H= 336:15 +CPU= 25.226
  09:45:27 STEP 1346 H= 336:30 +CPU= 24.550
  09:45:52 STEP 1347 H= 336:45 +CPU= 25.019
  09:46:16 STEP 1348 H= 337:00 +CPU= 24.877
  09:46:41 STEP 1349 H= 337:15 +CPU= 24.188
  09:47:05 STEP 1350 H= 337:30 +CPU= 24.495
  09:47:30 STEP 1351 H= 337:45 +CPU= 25.035
  09:47:55 STEP 1352 H= 338:00 +CPU= 24.864
  09:48:19 STEP 1353 H= 338:15 +CPU= 23.920
  09:48:44 STEP 1354 H= 338:30 +CPU= 25.158
  09:49:09 STEP 1355 H= 338:45 +CPU= 24.760
  09:49:39 STEP 1356 H= 339:00 +CPU= 30.014
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001356
  09:50:04 STEP 1357 H= 339:15 +CPU= 24.854
  09:50:28 STEP 1358 H= 339:30 +CPU= 24.572
  09:50:53 STEP 1359 H= 339:45 +CPU= 24.857
  09:51:18 STEP 1360 H= 340:00 +CPU= 24.347
  09:51:42 STEP 1361 H= 340:15 +CPU= 24.367
  09:52:07 STEP 1362 H= 340:30 +CPU= 25.117
  09:52:33 STEP 1363 H= 340:45 +CPU= 25.660
  09:52:58 STEP 1364 H= 341:00 +CPU= 25.129
  09:53:22 STEP 1365 H= 341:15 +CPU= 24.569
  09:53:48 STEP 1366 H= 341:30 +CPU= 25.264
  09:54:13 STEP 1367 H= 341:45 +CPU= 24.811
  09:54:43 STEP 1368 H= 342:00 +CPU= 30.370
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001368
  09:55:08 STEP 1369 H= 342:15 +CPU= 25.176
  09:55:33 STEP 1370 H= 342:30 +CPU= 24.718
  09:55:58 STEP 1371 H= 342:45 +CPU= 25.096
  09:56:23 STEP 1372 H= 343:00 +CPU= 25.047
  09:56:48 STEP 1373 H= 343:15 +CPU= 25.023
  09:57:12 STEP 1374 H= 343:30 +CPU= 24.065
  09:57:36 STEP 1375 H= 343:45 +CPU= 24.210
  09:58:01 STEP 1376 H= 344:00 +CPU= 25.046
  09:58:27 STEP 1377 H= 344:15 +CPU= 25.415
  09:58:52 STEP 1378 H= 344:30 +CPU= 24.910
  09:59:17 STEP 1379 H= 344:45 +CPU= 24.821
  09:59:46 STEP 1380 H= 345:00 +CPU= 29.875
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001380
  10:00:11 STEP 1381 H= 345:15 +CPU= 24.471
  10:00:35 STEP 1382 H= 345:30 +CPU= 24.325
  10:01:00 STEP 1383 H= 345:45 +CPU= 24.415
  10:01:24 STEP 1384 H= 346:00 +CPU= 24.805
  10:01:49 STEP 1385 H= 346:15 +CPU= 24.835
  10:02:14 STEP 1386 H= 346:30 +CPU= 24.812
  10:02:38 STEP 1387 H= 346:45 +CPU= 23.793
  10:03:02 STEP 1388 H= 347:00 +CPU= 24.421
  10:03:28 STEP 1389 H= 347:15 +CPU= 25.400
  10:03:53 STEP 1390 H= 347:30 +CPU= 25.280
  10:04:18 STEP 1391 H= 347:45 +CPU= 25.025
  10:04:53 STEP 1392 H= 348:00 +CPU= 35.094
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001392
  10:05:18 STEP 1393 H= 348:15 +CPU= 25.152
  10:05:44 STEP 1394 H= 348:30 +CPU= 25.187
  10:06:09 STEP 1395 H= 348:45 +CPU= 25.394
  10:06:34 STEP 1396 H= 349:00 +CPU= 25.380
  10:06:59 STEP 1397 H= 349:15 +CPU= 24.853
  10:07:24 STEP 1398 H= 349:30 +CPU= 25.206
  10:07:50 STEP 1399 H= 349:45 +CPU= 25.127
  10:08:14 STEP 1400 H= 350:00 +CPU= 24.628
  10:08:39 STEP 1401 H= 350:15 +CPU= 24.514
  10:09:04 STEP 1402 H= 350:30 +CPU= 25.529
  10:09:29 STEP 1403 H= 350:45 +CPU= 24.993
  10:09:59 STEP 1404 H= 351:00 +CPU= 29.568
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001404
  10:10:23 STEP 1405 H= 351:15 +CPU= 24.148
  10:10:48 STEP 1406 H= 351:30 +CPU= 24.560
  10:11:12 STEP 1407 H= 351:45 +CPU= 24.342
  10:11:37 STEP 1408 H= 352:00 +CPU= 25.064
  10:12:02 STEP 1409 H= 352:15 +CPU= 25.048
  10:12:26 STEP 1410 H= 352:30 +CPU= 23.935
  10:12:50 STEP 1411 H= 352:45 +CPU= 24.327
  10:13:16 STEP 1412 H= 353:00 +CPU= 25.221
  10:13:42 STEP 1413 H= 353:15 +CPU= 26.044
  10:14:07 STEP 1414 H= 353:30 +CPU= 25.191
  10:14:31 STEP 1415 H= 353:45 +CPU= 24.022
  10:15:01 STEP 1416 H= 354:00 +CPU= 30.073
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001416
  10:15:25 STEP 1417 H= 354:15 +CPU= 24.332
  10:15:50 STEP 1418 H= 354:30 +CPU= 24.307
  10:16:15 STEP 1419 H= 354:45 +CPU= 25.624
  10:16:40 STEP 1420 H= 355:00 +CPU= 25.280
  10:17:06 STEP 1421 H= 355:15 +CPU= 25.014
  10:17:32 STEP 1422 H= 355:30 +CPU= 26.190
  10:17:57 STEP 1423 H= 355:45 +CPU= 25.599
  10:18:23 STEP 1424 H= 356:00 +CPU= 25.730
  10:18:49 STEP 1425 H= 356:15 +CPU= 25.524
  10:19:15 STEP 1426 H= 356:30 +CPU= 25.938
  10:19:40 STEP 1427 H= 356:45 +CPU= 25.162
  10:20:10 STEP 1428 H= 357:00 +CPU= 30.521
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001428
  10:20:35 STEP 1429 H= 357:15 +CPU= 24.636
  10:21:00 STEP 1430 H= 357:30 +CPU= 25.137
  10:21:25 STEP 1431 H= 357:45 +CPU= 25.406
  10:21:51 STEP 1432 H= 358:00 +CPU= 25.493
  10:22:16 STEP 1433 H= 358:15 +CPU= 24.780
  10:22:40 STEP 1434 H= 358:30 +CPU= 24.663
  10:23:05 STEP 1435 H= 358:45 +CPU= 24.264
  10:23:27 STEP 1436 H= 359:00 +CPU= 22.833
  10:23:49 STEP 1437 H= 359:15 +CPU= 21.698
  10:24:12 STEP 1438 H= 359:30 +CPU= 22.486
  10:24:34 STEP 1439 H= 359:45 +CPU= 22.593
  10:25:03 STEP 1440 H= 360:00 +CPU= 28.953
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.6078939578E-21      I      -0.1883707878E-05      I
ICloud fractionI      0.0000000000E+00      I       0.5807484714E-01      I
I  Relat. Hum. I      0.4393028183E+00      I       0.4833727096E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.6248961005E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    323723668 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS539539.4 Mbytes/s, TOTAL TIME=    0.0074( 8.1%)
 10:25:03 STEP 1440 H= 360:00 +CPU= 28.953
 END CNT3
 NSTEP =  1440 CNT0   000000000  20.328  20.328  20.340   0.999               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        11440/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    203    0       46      1044068    11440 ************    24.1     1.2    22.9     1.7    42.3
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 8701392.1 8701392.1    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 8698819.6 8698819.6     99.97      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  417    4056.7    4056.7     19.44      0.00
   9 SPCM     - SPECTRAL COMP.                     384    1140.0    1140.0      5.03      0.00
  10 SCAN2M - PHYSICS                              385    9377.7    9377.7     41.49      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 32    1079.3    1079.3      0.40      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 17      63.1      63.1      0.01      0.00
  17 GRIDFPOS IN CNT4                               32      27.2      27.2      0.01      0.00
  27 RESTART FILES - WRITING                         8    4317.0    4317.0      0.40      0.00
  28 RESTART FILES - READING                         1    2140.9    2140.9      0.02      0.00
  29 SU4FPOS IN CNT4                                32       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                32    4250.1    4250.1      1.56      0.00
  31 POSDDH IN STEPO                                 2       5.1       5.1      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    385    4698.9    4698.9     20.79      0.00
  39 SU0YOMB                                         1     271.9     271.9      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    385      27.6      27.6      0.12      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              384     595.9     595.9      2.63      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              384      70.7      70.7      0.31      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 17       0.6       0.6      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      836     237.2     237.2      2.28      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       800     142.6     142.6      1.31      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        800      76.7      76.7      0.71      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       836     100.7     100.7      0.97      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      22.4      22.4      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            836      93.4      93.4      0.90      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            800      56.2      56.2      0.52      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            836      87.1      87.1      0.84      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            800      47.8      47.8      0.44      0.00
 400 GSTATS                                      45508       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 42288       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     8701.4 CPU TIME    8698.6 VECTOR TIME     8698.6


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      149.0
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/10/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/10/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284192/ICMGGh7zg+001440
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0_r1726112566_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
10:28:30 (5468): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0_r1726112566_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0_r1726112566_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0_r1726112566_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0_r1726112566_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0_r1726112566_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Jun 2024 16:23:00 1523278 22435772 oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0 1,295,100 37,175 0.0287
09 Jun 2024 16:23:00 1523278 22435772 oifs_43r3_bl_a1tp_2016092300_20_1017_12284192_0 864,000 25,576 0.0296


©2024 cpdn.org