climateprediction.net home page
Task 22435872

Task 22435872

Name oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0
Workunit 12284289
Created 7 Jun 2024, 14:24:23 UTC
Sent 7 Jun 2024, 14:30:13 UTC
Report deadline 6 Aug 2024, 14:30:13 UTC
Received 9 Jun 2024, 16:21:50 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1523278
Run time 10 hours 31 min 51 sec
CPU time 10 hours 27 min 46 sec
Validate state Invalid
Credit 1,318.46
Device peak FLOPS 8.64 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,001.19 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,578.80 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
g to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001020
  14:09:54 STEP 1021 H= 255:15 +CPU= 25.075
  14:10:20 STEP 1022 H= 255:30 +CPU= 25.194
  14:10:45 STEP 1023 H= 255:45 +CPU= 25.452
  14:11:10 STEP 1024 H= 256:00 +CPU= 25.804
  14:11:36 STEP 1025 H= 256:15 +CPU= 25.624
  14:12:02 STEP 1026 H= 256:30 +CPU= 25.495
  14:12:27 STEP 1027 H= 256:45 +CPU= 26.077
  14:12:55 STEP 1028 H= 257:00 +CPU= 27.288
  14:13:21 STEP 1029 H= 257:15 +CPU= 25.952
  14:13:46 STEP 1030 H= 257:30 +CPU= 26.088
  14:14:12 STEP 1031 H= 257:45 +CPU= 24.652
  14:14:42 STEP 1032 H= 258:00 +CPU= 30.643
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001032
  14:15:08 STEP 1033 H= 258:15 +CPU= 25.619
  14:15:34 STEP 1034 H= 258:30 +CPU= 25.500
  14:15:58 STEP 1035 H= 258:45 +CPU= 25.247
  14:16:27 STEP 1036 H= 259:00 +CPU= 28.042
  14:16:55 STEP 1037 H= 259:15 +CPU= 28.131
  14:17:24 STEP 1038 H= 259:30 +CPU= 28.302
  14:17:53 STEP 1039 H= 259:45 +CPU= 28.834
  14:18:22 STEP 1040 H= 260:00 +CPU= 29.257
  14:18:48 STEP 1041 H= 260:15 +CPU= 26.443
  14:19:12 STEP 1042 H= 260:30 +CPU= 25.353
  14:19:39 STEP 1043 H= 260:45 +CPU= 26.609
  14:20:10 STEP 1044 H= 261:00 +CPU= 30.736
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001044
  14:20:36 STEP 1045 H= 261:15 +CPU= 26.648
  14:21:04 STEP 1046 H= 261:30 +CPU= 26.043
  14:21:30 STEP 1047 H= 261:45 +CPU= 26.255
  14:21:55 STEP 1048 H= 262:00 +CPU= 25.820
  14:22:22 STEP 1049 H= 262:15 +CPU= 26.894
  14:22:50 STEP 1050 H= 262:30 +CPU= 28.539
  14:23:19 STEP 1051 H= 262:45 +CPU= 28.418
  14:23:48 STEP 1052 H= 263:00 +CPU= 28.869
  14:24:17 STEP 1053 H= 263:15 +CPU= 28.611
  14:24:45 STEP 1054 H= 263:30 +CPU= 28.021
  14:25:14 STEP 1055 H= 263:45 +CPU= 29.098
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.635] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4411MB (heap), 5021MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.636] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.636] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.636] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.637] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.637] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 4411 MB (maxheap), 5021 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.638] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142541:1717849541:78901.639] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142542:1717849542:78902.639] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Dinger7:1:1:94111:94111] [20240608:142542:1717849542:78902.639] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000116A671  cloudsc_                 1174  cloudsc.F90
oifs_43r3_model.e  000000000115C847  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  0000000000FBB064  callpar_                 1223  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1we
(argv4) batchid: 1017
(argv5) wuid: 12284289
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7d7565cb27d43e53583013dd3fa8d5e2 to: /var/lib/boinc-client/slots/7/oifs_43r3_bl_a1we_2016092300_20_1017_12284289.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_bl_a1we_2016092300_20_1017_12284289.zip
ic_ancil_file: ic_ancil_12284289
ifsdata_file: ifsdata_12284289
climate_data_file: clim_data_12284289
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/7/ic_ancil_12284289.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12284289.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_37c6dc3fb0eaa61f8d8105a33784ea2f to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12284289.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12284289.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12284289.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12284289.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12284289.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12284289.xml
-- Model is restarting --
Adjusting last_iter, 1056, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12284289.xml
last_cpu_time: 28472
upload_file_number: 10
last_iter: 1057
last_upload: 864000
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0_r1116515662
The child process has been launched with process id: 5599
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger7:1:1:5599:5599] [20240609:080416:1717913056:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0_r1116515662_10.zip
  08:04:31 STEP 1056 H= 264:00 +CPU= 12.435
Uploading the intermediate file: upload_file_10.zip
  08:04:43 STEP 1057 H= 264:15 +CPU= 11.521
  08:04:55 STEP 1058 H= 264:30 +CPU= 11.685
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001056
  08:05:06 STEP 1059 H= 264:45 +CPU= 11.799
  08:05:18 STEP 1060 H= 265:00 +CPU= 11.946
  08:05:30 STEP 1061 H= 265:15 +CPU= 11.575
  08:05:42 STEP 1062 H= 265:30 +CPU= 11.942
  08:05:54 STEP 1063 H= 265:45 +CPU= 11.984
  08:06:06 STEP 1064 H= 266:00 +CPU= 12.419
  08:06:18 STEP 1065 H= 266:15 +CPU= 11.946
  08:06:30 STEP 1066 H= 266:30 +CPU= 12.020
  08:06:43 STEP 1067 H= 266:45 +CPU= 12.164
  08:06:58 STEP 1068 H= 267:00 +CPU= 15.678
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001068
  08:07:10 STEP 1069 H= 267:15 +CPU= 11.945
  08:07:22 STEP 1070 H= 267:30 +CPU= 11.923
  08:07:34 STEP 1071 H= 267:45 +CPU= 11.905
  08:07:46 STEP 1072 H= 268:00 +CPU= 12.377
  08:07:59 STEP 1073 H= 268:15 +CPU= 12.189
  08:08:11 STEP 1074 H= 268:30 +CPU= 11.977
  08:08:22 STEP 1075 H= 268:45 +CPU= 11.779
  08:08:35 STEP 1076 H= 269:00 +CPU= 12.418
  08:08:47 STEP 1077 H= 269:15 +CPU= 11.994
  08:08:59 STEP 1078 H= 269:30 +CPU= 11.980
  08:09:11 STEP 1079 H= 269:45 +CPU= 12.008
  08:09:26 STEP 1080 H= 270:00 +CPU= 15.518
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001080
  08:09:38 STEP 1081 H= 270:15 +CPU= 11.914
  08:09:50 STEP 1082 H= 270:30 +CPU= 12.045
  08:10:02 STEP 1083 H= 270:45 +CPU= 11.884
  08:10:15 STEP 1084 H= 271:00 +CPU= 12.745
  08:10:27 STEP 1085 H= 271:15 +CPU= 12.246
  08:10:39 STEP 1086 H= 271:30 +CPU= 12.192
  08:10:52 STEP 1087 H= 271:45 +CPU= 12.540
  08:11:04 STEP 1088 H= 272:00 +CPU= 12.774
  08:11:17 STEP 1089 H= 272:15 +CPU= 12.697
  08:11:30 STEP 1090 H= 272:30 +CPU= 12.835
  08:11:43 STEP 1091 H= 272:45 +CPU= 13.306
  08:12:00 STEP 1092 H= 273:00 +CPU= 16.678
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001092
  08:12:13 STEP 1093 H= 273:15 +CPU= 12.919
  08:12:26 STEP 1094 H= 273:30 +CPU= 13.274
  08:12:39 STEP 1095 H= 273:45 +CPU= 13.172
  08:12:53 STEP 1096 H= 274:00 +CPU= 13.605
  08:13:06 STEP 1097 H= 274:15 +CPU= 13.273
  08:13:20 STEP 1098 H= 274:30 +CPU= 13.353
  08:13:33 STEP 1099 H= 274:45 +CPU= 13.426
  08:13:47 STEP 1100 H= 275:00 +CPU= 13.660
  08:14:01 STEP 1101 H= 275:15 +CPU= 13.021
  08:14:15 STEP 1102 H= 275:30 +CPU= 13.956
  08:14:29 STEP 1103 H= 275:45 +CPU= 14.091
  08:14:50 STEP 1104 H= 276:00 +CPU= 21.879
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001104
  08:15:05 STEP 1105 H= 276:15 +CPU= 14.959
  08:15:20 STEP 1106 H= 276:30 +CPU= 15.204
  08:15:37 STEP 1107 H= 276:45 +CPU= 16.059
  08:15:54 STEP 1108 H= 277:00 +CPU= 16.830
  08:16:11 STEP 1109 H= 277:15 +CPU= 17.152
  08:16:28 STEP 1110 H= 277:30 +CPU= 17.765
  08:16:48 STEP 1111 H= 277:45 +CPU= 19.498
  08:17:06 STEP 1112 H= 278:00 +CPU= 18.716
  08:17:28 STEP 1113 H= 278:15 +CPU= 21.045
  08:17:49 STEP 1114 H= 278:30 +CPU= 20.995
  08:18:10 STEP 1115 H= 278:45 +CPU= 21.849
  08:18:41 STEP 1116 H= 279:00 +CPU= 30.396
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001116
  08:19:07 STEP 1117 H= 279:15 +CPU= 26.099
  08:19:32 STEP 1118 H= 279:30 +CPU= 24.776
  08:19:56 STEP 1119 H= 279:45 +CPU= 24.557
  08:20:24 STEP 1120 H= 280:00 +CPU= 27.180
  08:20:52 STEP 1121 H= 280:15 +CPU= 27.828
  08:21:18 STEP 1122 H= 280:30 +CPU= 26.783
  08:21:45 STEP 1123 H= 280:45 +CPU= 26.886
  08:22:12 STEP 1124 H= 281:00 +CPU= 27.942
  08:22:40 STEP 1125 H= 281:15 +CPU= 27.306
  08:23:05 STEP 1126 H= 281:30 +CPU= 25.115
  08:23:31 STEP 1127 H= 281:45 +CPU= 25.712
  08:24:03 STEP 1128 H= 282:00 +CPU= 32.023
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001128
  08:24:30 STEP 1129 H= 282:15 +CPU= 26.975
  08:24:55 STEP 1130 H= 282:30 +CPU= 25.486
  08:25:23 STEP 1131 H= 282:45 +CPU= 27.283
  08:25:49 STEP 1132 H= 283:00 +CPU= 26.239
  08:26:15 STEP 1133 H= 283:15 +CPU= 26.446
  08:26:42 STEP 1134 H= 283:30 +CPU= 26.938
  08:27:08 STEP 1135 H= 283:45 +CPU= 25.943
  08:27:34 STEP 1136 H= 284:00 +CPU= 26.208
  08:28:00 STEP 1137 H= 284:15 +CPU= 25.285
  08:28:25 STEP 1138 H= 284:30 +CPU= 25.211
  08:28:50 STEP 1139 H= 284:45 +CPU= 25.064
  08:29:21 STEP 1140 H= 285:00 +CPU= 30.830
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001140
  08:29:46 STEP 1141 H= 285:15 +CPU= 24.927
  08:30:11 STEP 1142 H= 285:30 +CPU= 25.068
  08:30:36 STEP 1143 H= 285:45 +CPU= 25.029
  08:31:02 STEP 1144 H= 286:00 +CPU= 25.676
  08:31:27 STEP 1145 H= 286:15 +CPU= 25.519
  08:31:53 STEP 1146 H= 286:30 +CPU= 25.460
  08:32:17 STEP 1147 H= 286:45 +CPU= 24.761
  08:32:43 STEP 1148 H= 287:00 +CPU= 25.465
  08:33:08 STEP 1149 H= 287:15 +CPU= 25.160
  08:33:33 STEP 1150 H= 287:30 +CPU= 24.611
  08:33:58 STEP 1151 H= 287:45 +CPU= 25.807
  08:34:34 STEP 1152 H= 288:00 +CPU= 35.120
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001152
  08:34:59 STEP 1153 H= 288:15 +CPU= 25.417
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001152
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0_r1116515662_11.zip
Uploading the intermediate file: upload_file_11.zip
  08:35:24 STEP 1154 H= 288:30 +CPU= 25.409
  08:35:50 STEP 1155 H= 288:45 +CPU= 25.420
  08:36:16 STEP 1156 H= 289:00 +CPU= 25.899
  08:36:41 STEP 1157 H= 289:15 +CPU= 25.547
  08:37:07 STEP 1158 H= 289:30 +CPU= 25.517
  08:37:32 STEP 1159 H= 289:45 +CPU= 25.044
  08:37:58 STEP 1160 H= 290:00 +CPU= 26.603
  08:38:26 STEP 1161 H= 290:15 +CPU= 26.998
  08:38:52 STEP 1162 H= 290:30 +CPU= 26.375
  08:39:17 STEP 1163 H= 290:45 +CPU= 24.698
  08:39:49 STEP 1164 H= 291:00 +CPU= 32.115
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001164
  08:40:15 STEP 1165 H= 291:15 +CPU= 25.845
  08:40:40 STEP 1166 H= 291:30 +CPU= 24.997
  08:41:05 STEP 1167 H= 291:45 +CPU= 25.606
  08:41:31 STEP 1168 H= 292:00 +CPU= 25.977
  08:41:57 STEP 1169 H= 292:15 +CPU= 25.380
  08:42:22 STEP 1170 H= 292:30 +CPU= 24.924
  08:42:47 STEP 1171 H= 292:45 +CPU= 25.559
  08:43:12 STEP 1172 H= 293:00 +CPU= 25.148
  08:43:37 STEP 1173 H= 293:15 +CPU= 24.765
  08:44:02 STEP 1174 H= 293:30 +CPU= 24.926
  08:44:27 STEP 1175 H= 293:45 +CPU= 25.413
  08:44:58 STEP 1176 H= 294:00 +CPU= 30.685
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001176
  08:45:23 STEP 1177 H= 294:15 +CPU= 25.245
  08:45:48 STEP 1178 H= 294:30 +CPU= 24.855
  08:46:13 STEP 1179 H= 294:45 +CPU= 25.126
  08:46:39 STEP 1180 H= 295:00 +CPU= 25.412
  08:47:04 STEP 1181 H= 295:15 +CPU= 24.967
  08:47:29 STEP 1182 H= 295:30 +CPU= 25.078
  08:47:54 STEP 1183 H= 295:45 +CPU= 25.172
  08:48:19 STEP 1184 H= 296:00 +CPU= 25.055
  08:48:45 STEP 1185 H= 296:15 +CPU= 25.946
  08:49:11 STEP 1186 H= 296:30 +CPU= 25.724
  08:49:36 STEP 1187 H= 296:45 +CPU= 25.621
  08:50:07 STEP 1188 H= 297:00 +CPU= 30.391
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001188
  08:50:32 STEP 1189 H= 297:15 +CPU= 25.538
  08:50:58 STEP 1190 H= 297:30 +CPU= 25.525
  08:51:23 STEP 1191 H= 297:45 +CPU= 25.202
  08:51:49 STEP 1192 H= 298:00 +CPU= 25.540
  08:52:14 STEP 1193 H= 298:15 +CPU= 25.357
  08:52:39 STEP 1194 H= 298:30 +CPU= 25.532
  08:53:05 STEP 1195 H= 298:45 +CPU= 25.522
  08:53:30 STEP 1196 H= 299:00 +CPU= 24.632
  08:53:55 STEP 1197 H= 299:15 +CPU= 24.973
  08:54:21 STEP 1198 H= 299:30 +CPU= 25.883
  08:54:46 STEP 1199 H= 299:45 +CPU= 25.726
  08:55:22 STEP 1200 H= 300:00 +CPU= 35.875
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001200
  08:55:47 STEP 1201 H= 300:15 +CPU= 25.111
  08:56:14 STEP 1202 H= 300:30 +CPU= 26.196
  08:56:39 STEP 1203 H= 300:45 +CPU= 25.162
  08:57:04 STEP 1204 H= 301:00 +CPU= 25.006
  08:57:29 STEP 1205 H= 301:15 +CPU= 25.718
  08:57:55 STEP 1206 H= 301:30 +CPU= 25.239
  08:58:20 STEP 1207 H= 301:45 +CPU= 25.673
  08:58:46 STEP 1208 H= 302:00 +CPU= 25.527
  08:59:12 STEP 1209 H= 302:15 +CPU= 25.979
  08:59:37 STEP 1210 H= 302:30 +CPU= 25.115
  09:00:02 STEP 1211 H= 302:45 +CPU= 24.864
  09:00:32 STEP 1212 H= 303:00 +CPU= 29.852
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001212
  09:00:57 STEP 1213 H= 303:15 +CPU= 25.111
  09:01:23 STEP 1214 H= 303:30 +CPU= 26.334
  09:01:48 STEP 1215 H= 303:45 +CPU= 25.149
  09:02:14 STEP 1216 H= 304:00 +CPU= 25.338
  09:02:39 STEP 1217 H= 304:15 +CPU= 25.094
  09:03:04 STEP 1218 H= 304:30 +CPU= 24.808
  09:03:28 STEP 1219 H= 304:45 +CPU= 24.853
  09:03:53 STEP 1220 H= 305:00 +CPU= 24.947
  09:04:19 STEP 1221 H= 305:15 +CPU= 25.875
  09:04:44 STEP 1222 H= 305:30 +CPU= 25.067
  09:05:09 STEP 1223 H= 305:45 +CPU= 24.601
  09:05:39 STEP 1224 H= 306:00 +CPU= 30.530
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001224
  09:06:05 STEP 1225 H= 306:15 +CPU= 25.732
  09:06:31 STEP 1226 H= 306:30 +CPU= 25.422
  09:06:55 STEP 1227 H= 306:45 +CPU= 24.662
  09:07:20 STEP 1228 H= 307:00 +CPU= 25.051
  09:07:45 STEP 1229 H= 307:15 +CPU= 24.790
  09:08:10 STEP 1230 H= 307:30 +CPU= 24.763
  09:08:35 STEP 1231 H= 307:45 +CPU= 24.969
  09:09:00 STEP 1232 H= 308:00 +CPU= 25.109
  09:09:26 STEP 1233 H= 308:15 +CPU= 25.530
  09:09:51 STEP 1234 H= 308:30 +CPU= 25.602
  09:10:17 STEP 1235 H= 308:45 +CPU= 25.428
  09:10:49 STEP 1236 H= 309:00 +CPU= 32.085
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001236
  09:11:14 STEP 1237 H= 309:15 +CPU= 25.327
  09:11:40 STEP 1238 H= 309:30 +CPU= 25.794
  09:12:05 STEP 1239 H= 309:45 +CPU= 25.295
  09:12:32 STEP 1240 H= 310:00 +CPU= 26.402
  09:12:57 STEP 1241 H= 310:15 +CPU= 25.163
  09:13:22 STEP 1242 H= 310:30 +CPU= 25.471
  09:13:48 STEP 1243 H= 310:45 +CPU= 25.845
  09:14:15 STEP 1244 H= 311:00 +CPU= 26.709
  09:14:40 STEP 1245 H= 311:15 +CPU= 25.429
  09:15:07 STEP 1246 H= 311:30 +CPU= 26.482
  09:15:32 STEP 1247 H= 311:45 +CPU= 25.066
  09:16:08 STEP 1248 H= 312:00 +CPU= 35.807
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001248
  09:16:33 STEP 1249 H= 312:15 +CPU= 25.387
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001248
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0_r1116515662_12.zip
Uploading the intermediate file: upload_file_12.zip
  09:16:58 STEP 1250 H= 312:30 +CPU= 25.311
  09:17:23 STEP 1251 H= 312:45 +CPU= 25.110
  09:17:48 STEP 1252 H= 313:00 +CPU= 25.025
  09:18:13 STEP 1253 H= 313:15 +CPU= 25.036
  09:18:38 STEP 1254 H= 313:30 +CPU= 24.317
  09:19:03 STEP 1255 H= 313:45 +CPU= 24.754
  09:19:28 STEP 1256 H= 314:00 +CPU= 25.555
  09:19:54 STEP 1257 H= 314:15 +CPU= 25.601
  09:20:20 STEP 1258 H= 314:30 +CPU= 25.781
  09:20:45 STEP 1259 H= 314:45 +CPU= 25.945
  09:21:17 STEP 1260 H= 315:00 +CPU= 31.251
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001260
  09:21:42 STEP 1261 H= 315:15 +CPU= 25.043
  09:22:07 STEP 1262 H= 315:30 +CPU= 25.616
  09:22:33 STEP 1263 H= 315:45 +CPU= 25.367
  09:22:58 STEP 1264 H= 316:00 +CPU= 25.660
  09:23:24 STEP 1265 H= 316:15 +CPU= 25.624
  09:23:48 STEP 1266 H= 316:30 +CPU= 24.294
  09:24:13 STEP 1267 H= 316:45 +CPU= 24.725
  09:24:39 STEP 1268 H= 317:00 +CPU= 26.326
  09:25:05 STEP 1269 H= 317:15 +CPU= 25.932
  09:25:30 STEP 1270 H= 317:30 +CPU= 25.057
  09:25:55 STEP 1271 H= 317:45 +CPU= 24.699
  09:26:27 STEP 1272 H= 318:00 +CPU= 31.600
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001272
  09:26:52 STEP 1273 H= 318:15 +CPU= 25.338
  09:27:17 STEP 1274 H= 318:30 +CPU= 24.863
  09:27:42 STEP 1275 H= 318:45 +CPU= 24.883
  09:28:07 STEP 1276 H= 319:00 +CPU= 25.470
  09:28:33 STEP 1277 H= 319:15 +CPU= 25.384
  09:28:58 STEP 1278 H= 319:30 +CPU= 24.853
  09:29:23 STEP 1279 H= 319:45 +CPU= 25.043
  09:29:49 STEP 1280 H= 320:00 +CPU= 25.882
  09:30:13 STEP 1281 H= 320:15 +CPU= 24.481
  09:30:37 STEP 1282 H= 320:30 +CPU= 24.068
  09:31:02 STEP 1283 H= 320:45 +CPU= 24.939
  09:31:33 STEP 1284 H= 321:00 +CPU= 30.547
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001284
  09:31:58 STEP 1285 H= 321:15 +CPU= 24.920
  09:32:23 STEP 1286 H= 321:30 +CPU= 25.043
  09:32:47 STEP 1287 H= 321:45 +CPU= 24.597
  09:33:12 STEP 1288 H= 322:00 +CPU= 24.973
  09:33:37 STEP 1289 H= 322:15 +CPU= 24.439
  09:34:01 STEP 1290 H= 322:30 +CPU= 24.601
  09:34:26 STEP 1291 H= 322:45 +CPU= 25.278
  09:34:52 STEP 1292 H= 323:00 +CPU= 25.501
  09:35:16 STEP 1293 H= 323:15 +CPU= 23.959
  09:35:41 STEP 1294 H= 323:30 +CPU= 24.977
  09:36:06 STEP 1295 H= 323:45 +CPU= 25.498
  09:36:42 STEP 1296 H= 324:00 +CPU= 35.925
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001296
  09:37:08 STEP 1297 H= 324:15 +CPU= 25.322
  09:37:34 STEP 1298 H= 324:30 +CPU= 25.805
  09:37:59 STEP 1299 H= 324:45 +CPU= 25.595
  09:38:25 STEP 1300 H= 325:00 +CPU= 25.477
  09:38:50 STEP 1301 H= 325:15 +CPU= 25.531
  09:39:16 STEP 1302 H= 325:30 +CPU= 25.497
  09:39:41 STEP 1303 H= 325:45 +CPU= 25.480
  09:40:06 STEP 1304 H= 326:00 +CPU= 25.149
  09:40:32 STEP 1305 H= 326:15 +CPU= 25.577
  09:40:57 STEP 1306 H= 326:30 +CPU= 25.258
  09:41:22 STEP 1307 H= 326:45 +CPU= 25.205
  09:41:53 STEP 1308 H= 327:00 +CPU= 30.431
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001308
  09:42:18 STEP 1309 H= 327:15 +CPU= 24.986
  09:42:43 STEP 1310 H= 327:30 +CPU= 25.156
  09:43:08 STEP 1311 H= 327:45 +CPU= 24.814
  09:43:34 STEP 1312 H= 328:00 +CPU= 25.756
  09:43:59 STEP 1313 H= 328:15 +CPU= 25.338
  09:44:24 STEP 1314 H= 328:30 +CPU= 25.442
  09:44:50 STEP 1315 H= 328:45 +CPU= 25.547
  09:45:15 STEP 1316 H= 329:00 +CPU= 25.464
  09:45:40 STEP 1317 H= 329:15 +CPU= 24.310
  09:46:04 STEP 1318 H= 329:30 +CPU= 24.762
  09:46:29 STEP 1319 H= 329:45 +CPU= 24.986
  09:47:00 STEP 1320 H= 330:00 +CPU= 30.384
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001320
  09:47:25 STEP 1321 H= 330:15 +CPU= 24.820
  09:47:50 STEP 1322 H= 330:30 +CPU= 25.625
  09:48:16 STEP 1323 H= 330:45 +CPU= 25.447
  09:48:42 STEP 1324 H= 331:00 +CPU= 26.000
  09:49:07 STEP 1325 H= 331:15 +CPU= 25.310
  09:49:32 STEP 1326 H= 331:30 +CPU= 24.874
  09:49:58 STEP 1327 H= 331:45 +CPU= 25.564
  09:50:23 STEP 1328 H= 332:00 +CPU= 25.480
  09:50:48 STEP 1329 H= 332:15 +CPU= 24.893
  09:51:14 STEP 1330 H= 332:30 +CPU= 26.194
  09:51:39 STEP 1331 H= 332:45 +CPU= 25.046
  09:52:10 STEP 1332 H= 333:00 +CPU= 30.498
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001332
  09:52:35 STEP 1333 H= 333:15 +CPU= 25.123
  09:52:59 STEP 1334 H= 333:30 +CPU= 23.958
  09:53:24 STEP 1335 H= 333:45 +CPU= 25.133
  09:53:50 STEP 1336 H= 334:00 +CPU= 25.814
  09:54:15 STEP 1337 H= 334:15 +CPU= 25.535
  09:54:40 STEP 1338 H= 334:30 +CPU= 25.091
  09:55:05 STEP 1339 H= 334:45 +CPU= 24.990
  09:55:32 STEP 1340 H= 335:00 +CPU= 26.566
  09:55:57 STEP 1341 H= 335:15 +CPU= 25.009
  09:56:22 STEP 1342 H= 335:30 +CPU= 25.517
  09:56:48 STEP 1343 H= 335:45 +CPU= 25.484
  09:57:23 STEP 1344 H= 336:00 +CPU= 35.037
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001344
  09:57:48 STEP 1345 H= 336:15 +CPU= 25.012
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001344
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0_r1116515662_13.zip
Uploading the intermediate file: upload_file_13.zip
  09:58:14 STEP 1346 H= 336:30 +CPU= 25.797
  09:58:40 STEP 1347 H= 336:45 +CPU= 25.764
  09:59:05 STEP 1348 H= 337:00 +CPU= 25.840
  09:59:30 STEP 1349 H= 337:15 +CPU= 24.977
  09:59:56 STEP 1350 H= 337:30 +CPU= 25.192
  10:00:21 STEP 1351 H= 337:45 +CPU= 25.757
  10:00:48 STEP 1352 H= 338:00 +CPU= 26.226
  10:01:13 STEP 1353 H= 338:15 +CPU= 25.198
  10:01:38 STEP 1354 H= 338:30 +CPU= 25.546
  10:02:04 STEP 1355 H= 338:45 +CPU= 25.442
  10:02:34 STEP 1356 H= 339:00 +CPU= 30.696
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001356
  10:03:00 STEP 1357 H= 339:15 +CPU= 25.030
  10:03:25 STEP 1358 H= 339:30 +CPU= 25.219
  10:03:50 STEP 1359 H= 339:45 +CPU= 24.840
  10:04:16 STEP 1360 H= 340:00 +CPU= 25.958
  10:04:41 STEP 1361 H= 340:15 +CPU= 25.104
  10:05:06 STEP 1362 H= 340:30 +CPU= 24.995
  10:05:31 STEP 1363 H= 340:45 +CPU= 25.403
  10:05:57 STEP 1364 H= 341:00 +CPU= 26.098
  10:06:23 STEP 1365 H= 341:15 +CPU= 26.108
  10:06:49 STEP 1366 H= 341:30 +CPU= 25.570
  10:07:14 STEP 1367 H= 341:45 +CPU= 25.486
  10:07:44 STEP 1368 H= 342:00 +CPU= 29.924
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001368
  10:08:10 STEP 1369 H= 342:15 +CPU= 25.573
  10:08:35 STEP 1370 H= 342:30 +CPU= 25.473
  10:09:01 STEP 1371 H= 342:45 +CPU= 25.180
  10:09:26 STEP 1372 H= 343:00 +CPU= 25.569
  10:09:51 STEP 1373 H= 343:15 +CPU= 24.781
  10:10:15 STEP 1374 H= 343:30 +CPU= 24.400
  10:10:40 STEP 1375 H= 343:45 +CPU= 24.722
  10:11:06 STEP 1376 H= 344:00 +CPU= 25.935
  10:11:31 STEP 1377 H= 344:15 +CPU= 25.222
  10:11:57 STEP 1378 H= 344:30 +CPU= 25.336
  10:12:21 STEP 1379 H= 344:45 +CPU= 24.907
  10:12:51 STEP 1380 H= 345:00 +CPU= 29.886
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001380
  10:13:16 STEP 1381 H= 345:15 +CPU= 25.099
  10:13:43 STEP 1382 H= 345:30 +CPU= 26.223
  10:14:08 STEP 1383 H= 345:45 +CPU= 25.081
  10:14:34 STEP 1384 H= 346:00 +CPU= 25.772
  10:14:59 STEP 1385 H= 346:15 +CPU= 25.365
  10:15:25 STEP 1386 H= 346:30 +CPU= 26.547
  10:15:51 STEP 1387 H= 346:45 +CPU= 25.080
  10:16:16 STEP 1388 H= 347:00 +CPU= 25.420
  10:16:41 STEP 1389 H= 347:15 +CPU= 25.357
  10:17:07 STEP 1390 H= 347:30 +CPU= 25.916
  10:17:33 STEP 1391 H= 347:45 +CPU= 25.942
  10:18:09 STEP 1392 H= 348:00 +CPU= 35.551
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001392
  10:18:35 STEP 1393 H= 348:15 +CPU= 25.882
  10:19:00 STEP 1394 H= 348:30 +CPU= 25.683
  10:19:26 STEP 1395 H= 348:45 +CPU= 25.887
  10:19:52 STEP 1396 H= 349:00 +CPU= 25.494
  10:20:17 STEP 1397 H= 349:15 +CPU= 25.434
  10:20:43 STEP 1398 H= 349:30 +CPU= 25.525
  10:21:08 STEP 1399 H= 349:45 +CPU= 25.414
  10:21:34 STEP 1400 H= 350:00 +CPU= 25.795
  10:21:59 STEP 1401 H= 350:15 +CPU= 25.350
  10:22:24 STEP 1402 H= 350:30 +CPU= 24.882
  10:22:50 STEP 1403 H= 350:45 +CPU= 25.396
  10:23:19 STEP 1404 H= 351:00 +CPU= 29.052
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001404
  10:23:41 STEP 1405 H= 351:15 +CPU= 22.326
  10:24:04 STEP 1406 H= 351:30 +CPU= 22.918
  10:24:26 STEP 1407 H= 351:45 +CPU= 22.186
  10:24:49 STEP 1408 H= 352:00 +CPU= 22.665
  10:25:11 STEP 1409 H= 352:15 +CPU= 22.038
  10:25:31 STEP 1410 H= 352:30 +CPU= 20.220
  10:25:51 STEP 1411 H= 352:45 +CPU= 19.938
  10:26:11 STEP 1412 H= 353:00 +CPU= 20.268
  10:26:31 STEP 1413 H= 353:15 +CPU= 19.750
  10:26:52 STEP 1414 H= 353:30 +CPU= 21.356
  10:27:16 STEP 1415 H= 353:45 +CPU= 23.216
  10:27:43 STEP 1416 H= 354:00 +CPU= 27.684
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001416
  10:28:06 STEP 1417 H= 354:15 +CPU= 23.211
  10:28:30 STEP 1418 H= 354:30 +CPU= 23.142
  10:28:53 STEP 1419 H= 354:45 +CPU= 23.856
  10:29:20 STEP 1420 H= 355:00 +CPU= 26.319
  10:29:45 STEP 1421 H= 355:15 +CPU= 25.012
  10:30:10 STEP 1422 H= 355:30 +CPU= 24.800
  10:30:33 STEP 1423 H= 355:45 +CPU= 23.675
  10:30:56 STEP 1424 H= 356:00 +CPU= 22.963
  10:31:18 STEP 1425 H= 356:15 +CPU= 22.082
  10:31:40 STEP 1426 H= 356:30 +CPU= 21.297
  10:32:02 STEP 1427 H= 356:45 +CPU= 22.737
  10:32:29 STEP 1428 H= 357:00 +CPU= 27.083
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001428
  10:32:52 STEP 1429 H= 357:15 +CPU= 22.510
  10:33:15 STEP 1430 H= 357:30 +CPU= 22.592
  10:33:36 STEP 1431 H= 357:45 +CPU= 21.626
  10:34:01 STEP 1432 H= 358:00 +CPU= 24.536
  10:34:26 STEP 1433 H= 358:15 +CPU= 25.356
  10:34:51 STEP 1434 H= 358:30 +CPU= 25.128
  10:35:18 STEP 1435 H= 358:45 +CPU= 26.575
  10:35:45 STEP 1436 H= 359:00 +CPU= 26.865
  10:36:11 STEP 1437 H= 359:15 +CPU= 25.829
  10:36:36 STEP 1438 H= 359:30 +CPU= 25.432
  10:37:01 STEP 1439 H= 359:45 +CPU= 25.198
  10:37:34 STEP 1440 H= 360:00 +CPU= 32.497
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1703474119E-20      I       0.1104738960E-04      I
ICloud fractionI      0.0000000000E+00      I       0.5987119936E-01      I
I  Relat. Hum. I      0.3172808058E+00      I       0.4084643562E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.9316097840E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    295538788 BYTES IN      7 RECORDS TRANSFERRED IN    0.0007 SECONDS422198.3 Mbytes/s, TOTAL TIME=    0.0087( 8.0%)
 10:37:34 STEP 1440 H= 360:00 +CPU= 32.497
 END CNT3
 NSTEP =  1440 CNT0   000000000  23.003  23.003  23.008   1.000               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        11440/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    203    0       46      1044068    11440 ************    24.9     1.2    23.6     1.7    43.8
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 9197329.5 9197329.5    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 9194719.8 9194719.8     99.97      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  417    4295.5    4295.5     19.48      0.00
   9 SPCM     - SPECTRAL COMP.                     384    1165.0    1165.0      4.86      0.00
  10 SCAN2M - PHYSICS                              385   10095.3   10095.3     42.26      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 32    1120.5    1120.5      0.39      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 17      66.6      66.6      0.01      0.00
  17 GRIDFPOS IN CNT4                               32      29.3      29.3      0.01      0.00
  27 RESTART FILES - WRITING                         8    4481.9    4481.9      0.39      0.00
  28 RESTART FILES - READING                         1    2161.2    2161.2      0.02      0.00
  29 SU4FPOS IN CNT4                                32       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                32    4345.1    4345.1      1.51      0.00
  31 POSDDH IN STEPO                                 2       2.3       2.3      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    385    4925.7    4925.7     20.62      0.00
  39 SU0YOMB                                         1     272.4     272.4      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    385      28.1      28.1      0.12      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              384     595.1     595.1      2.48      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              384      72.4      72.4      0.30      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 17       0.6       0.6      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      836     241.5     241.5      2.20      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       800     144.2     144.2      1.25      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        800      78.5      78.5      0.68      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       836     103.8     103.8      0.94      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      23.3      23.3      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            836      95.9      95.9      0.87      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            800      57.9      57.9      0.50      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            836      91.0      91.0      0.83      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            800      50.3      50.3      0.44      0.00
 400 GSTATS                                      45508       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 42288       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     9197.3 CPU TIME    9194.6 VECTOR TIME     9194.6


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      141.0
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/7/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/7/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284289/ICMGGh7zg+001440
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0_r1116515662_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
10:41:02 (5596): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0_r1116515662_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0_r1116515662_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0_r1116515662_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0_r1116515662_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0_r1116515662_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Jun 2024 16:23:00 1523278 22435872 oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0 1,295,100 37,666 0.0291
09 Jun 2024 16:23:00 1523278 22435872 oifs_43r3_bl_a1we_2016092300_20_1017_12284289_0 864,000 25,868 0.0299


©2024 climateprediction.net