<core_client_version>7.22.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
l_12292912
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/4/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/4/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_651068079fac398a28c088784343c8e0 to: /var/lib/boinc/slots/4/oifs_43r3_bl_a3jd_2016092300_20_1018_12292912.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/4/oifs_43r3_bl_a3jd_2016092300_20_1018_12292912.zip
ic_ancil_file: ic_ancil_12292912
ifsdata_file: ifsdata_12292912
climate_data_file: clim_data_12292912
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/4/ic_ancil_12292912.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/4/ic_ancil_12292912.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d57a568d0c430b89a90ffab3b11cb8d6 to: /var/lib/boinc/slots/4/ifsdata/ifsdata_12292912.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/4/ifsdata/ifsdata_12292912.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/4/159l_2/clim_data_12292912.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/4/159l_2/clim_data_12292912.zip
Checking for progress XML file: /var/lib/boinc/slots/4/progress_file_12292912.xml
Opened progress file ok : /var/lib/boinc/slots/4/progress_file_12292912.xml
-- Model is restarting --
Adjusting last_iter, 768, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/4/progress_file_12292912.xml
last_cpu_time: 14664
upload_file_number: 7
last_iter: 769
last_upload: 605700
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3jd_2016092300_20_1018_12292912_0_r1266141355
The child process has been launched with process id: 548988
Executing the command: /var/lib/boinc/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:548988:548988] [20240613:193201:1718299921:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000684
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000684
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000684
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000696
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000696
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000696
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000708
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000708
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000708
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000720
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000720
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000720
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000732
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000732
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000732
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000756
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000756
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000756
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3jd_2016092300_20_1018_12292912_0_r1266141355_7.zip
Uploading the intermediate file: upload_file_7.zip
19:32:23 STEP 768 H= 192:00 +CPU= 19.458
19:32:41 STEP 769 H= 192:15 +CPU= 18.227
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000768
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000768
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000768
19:32:59 STEP 770 H= 192:30 +CPU= 18.001
19:33:18 STEP 771 H= 192:45 +CPU= 18.448
19:33:37 STEP 772 H= 193:00 +CPU= 19.120
19:33:55 STEP 773 H= 193:15 +CPU= 18.610
19:34:14 STEP 774 H= 193:30 +CPU= 18.118
19:34:32 STEP 775 H= 193:45 +CPU= 18.073
19:34:51 STEP 776 H= 194:00 +CPU= 19.158
19:35:10 STEP 777 H= 194:15 +CPU= 18.567
19:35:27 STEP 778 H= 194:30 +CPU= 17.876
19:35:46 STEP 779 H= 194:45 +CPU= 18.808
19:36:11 STEP 780 H= 195:00 +CPU= 24.212
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000780
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000780
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000780
19:36:29 STEP 781 H= 195:15 +CPU= 18.501
19:36:48 STEP 782 H= 195:30 +CPU= 18.426
19:37:06 STEP 783 H= 195:45 +CPU= 18.765
19:37:27 STEP 784 H= 196:00 +CPU= 20.191
19:38:01 STEP 785 H= 196:15 +CPU= 18.607
19:38:19 STEP 786 H= 196:30 +CPU= 17.977
19:38:37 STEP 787 H= 196:45 +CPU= 18.201
19:38:57 STEP 788 H= 197:00 +CPU= 19.679
19:39:16 STEP 789 H= 197:15 +CPU= 18.548
19:39:34 STEP 790 H= 197:30 +CPU= 17.955
19:39:52 STEP 791 H= 197:45 +CPU= 18.181
19:40:16 STEP 792 H= 198:00 +CPU= 23.875
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000792
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000792
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000792
19:40:34 STEP 793 H= 198:15 +CPU= 18.451
19:40:53 STEP 794 H= 198:30 +CPU= 18.561
19:41:11 STEP 795 H= 198:45 +CPU= 17.711
19:41:29 STEP 796 H= 199:00 +CPU= 18.588
19:41:48 STEP 797 H= 199:15 +CPU= 18.604
19:42:06 STEP 798 H= 199:30 +CPU= 18.299
19:42:24 STEP 799 H= 199:45 +CPU= 17.958
19:42:43 STEP 800 H= 200:00 +CPU= 18.807
19:43:01 STEP 801 H= 200:15 +CPU= 18.155
19:43:20 STEP 802 H= 200:30 +CPU= 18.351
19:43:39 STEP 803 H= 200:45 +CPU= 18.784
19:44:03 STEP 804 H= 201:00 +CPU= 24.828
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000804
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000804
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000804
19:44:22 STEP 805 H= 201:15 +CPU= 18.420
19:44:40 STEP 806 H= 201:30 +CPU= 17.759
19:44:58 STEP 807 H= 201:45 +CPU= 18.102
19:45:17 STEP 808 H= 202:00 +CPU= 19.220
19:45:36 STEP 809 H= 202:15 +CPU= 18.402
19:45:54 STEP 810 H= 202:30 +CPU= 18.126
19:46:12 STEP 811 H= 202:45 +CPU= 17.931
19:46:31 STEP 812 H= 203:00 +CPU= 19.384
19:46:50 STEP 813 H= 203:15 +CPU= 18.463
19:47:08 STEP 814 H= 203:30 +CPU= 18.710
19:47:27 STEP 815 H= 203:45 +CPU= 18.997
19:47:54 STEP 816 H= 204:00 +CPU= 26.469
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000816
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000816
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000816
19:48:12 STEP 817 H= 204:15 +CPU= 18.443
19:48:31 STEP 818 H= 204:30 +CPU= 18.224
19:48:49 STEP 819 H= 204:45 +CPU= 18.150
19:49:08 STEP 820 H= 205:00 +CPU= 19.250
19:49:27 STEP 821 H= 205:15 +CPU= 18.932
19:49:46 STEP 822 H= 205:30 +CPU= 18.824
19:50:04 STEP 823 H= 205:45 +CPU= 18.385
19:50:24 STEP 824 H= 206:00 +CPU= 19.609
19:50:43 STEP 825 H= 206:15 +CPU= 18.966
19:51:02 STEP 826 H= 206:30 +CPU= 18.476
19:51:20 STEP 827 H= 206:45 +CPU= 18.443
19:51:44 STEP 828 H= 207:00 +CPU= 23.730
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000828
19:52:02 STEP 829 H= 207:15 +CPU= 18.286
19:52:20 STEP 830 H= 207:30 +CPU= 17.882
19:52:38 STEP 831 H= 207:45 +CPU= 17.805
19:52:57 STEP 832 H= 208:00 +CPU= 18.993
19:53:16 STEP 833 H= 208:15 +CPU= 18.847
19:53:34 STEP 834 H= 208:30 +CPU= 18.409
19:53:53 STEP 835 H= 208:45 +CPU= 18.623
19:54:12 STEP 836 H= 209:00 +CPU= 18.960
19:54:31 STEP 837 H= 209:15 +CPU= 18.708
19:54:49 STEP 838 H= 209:30 +CPU= 18.185
19:55:07 STEP 839 H= 209:45 +CPU= 18.140
19:55:31 STEP 840 H= 210:00 +CPU= 24.170
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000840
19:55:50 STEP 841 H= 210:15 +CPU= 18.479
19:56:08 STEP 842 H= 210:30 +CPU= 18.286
19:56:26 STEP 843 H= 210:45 +CPU= 18.214
19:56:45 STEP 844 H= 211:00 +CPU= 19.003
19:57:03 STEP 845 H= 211:15 +CPU= 18.019
19:57:22 STEP 846 H= 211:30 +CPU= 18.780
19:57:41 STEP 847 H= 211:45 +CPU= 18.291
19:58:00 STEP 848 H= 212:00 +CPU= 19.770
19:58:19 STEP 849 H= 212:15 +CPU= 18.278
19:58:37 STEP 850 H= 212:30 +CPU= 18.199
19:58:55 STEP 851 H= 212:45 +CPU= 18.261
19:59:19 STEP 852 H= 213:00 +CPU= 24.224
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000852
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000852
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000852
19:59:38 STEP 853 H= 213:15 +CPU= 18.423
19:59:56 STEP 854 H= 213:30 +CPU= 18.400
20:00:15 STEP 855 H= 213:45 +CPU= 18.132
20:00:34 STEP 856 H= 214:00 +CPU= 19.198
20:00:52 STEP 857 H= 214:15 +CPU= 18.130
20:01:10 STEP 858 H= 214:30 +CPU= 18.035
20:01:28 STEP 859 H= 214:45 +CPU= 18.255
20:01:48 STEP 860 H= 215:00 +CPU= 19.379
20:02:07 STEP 861 H= 215:15 +CPU= 18.765
20:02:25 STEP 862 H= 215:30 +CPU= 18.522
20:02:43 STEP 863 H= 215:45 +CPU= 18.162
20:03:10 STEP 864 H= 216:00 +CPU= 26.574
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000864
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000864
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000864
20:03:29 STEP 865 H= 216:15 +CPU= 18.833
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000852
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000852
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000852
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000864
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3jd_2016092300_20_1018_12292912_0_r1266141355_8.zip
Uploading the intermediate file: upload_file_8.zip
20:03:47 STEP 866 H= 216:30 +CPU= 18.291
20:04:05 STEP 867 H= 216:45 +CPU= 18.093
20:04:25 STEP 868 H= 217:00 +CPU= 19.337
20:04:44 STEP 869 H= 217:15 +CPU= 19.114
20:05:03 STEP 870 H= 217:30 +CPU= 18.809
20:05:21 STEP 871 H= 217:45 +CPU= 18.570
20:05:40 STEP 872 H= 218:00 +CPU= 19.096
20:05:59 STEP 873 H= 218:15 +CPU= 18.326
20:06:17 STEP 874 H= 218:30 +CPU= 18.480
20:06:35 STEP 875 H= 218:45 +CPU= 18.026
20:06:59 STEP 876 H= 219:00 +CPU= 24.103
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000876
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000876
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000876
20:07:18 STEP 877 H= 219:15 +CPU= 18.704
20:07:37 STEP 878 H= 219:30 +CPU= 18.596
20:07:56 STEP 879 H= 219:45 +CPU= 18.571
20:08:14 STEP 880 H= 220:00 +CPU= 18.862
20:08:32 STEP 881 H= 220:15 +CPU= 17.891
20:08:51 STEP 882 H= 220:30 +CPU= 18.149
20:09:09 STEP 883 H= 220:45 +CPU= 18.069
20:09:28 STEP 884 H= 221:00 +CPU= 19.508
20:09:47 STEP 885 H= 221:15 +CPU= 18.793
20:10:06 STEP 886 H= 221:30 +CPU= 18.903
20:10:25 STEP 887 H= 221:45 +CPU= 19.054
20:10:49 STEP 888 H= 222:00 +CPU= 24.288
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000888
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000888
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000888
20:11:07 STEP 889 H= 222:15 +CPU= 17.900
20:11:26 STEP 890 H= 222:30 +CPU= 18.346
20:11:44 STEP 891 H= 222:45 +CPU= 18.221
20:12:03 STEP 892 H= 223:00 +CPU= 19.397
20:12:22 STEP 893 H= 223:15 +CPU= 18.480
20:12:41 STEP 894 H= 223:30 +CPU= 18.836
20:13:00 STEP 895 H= 223:45 +CPU= 18.664
20:13:19 STEP 896 H= 224:00 +CPU= 19.295
20:13:37 STEP 897 H= 224:15 +CPU= 18.337
20:13:56 STEP 898 H= 224:30 +CPU= 18.734
20:14:15 STEP 899 H= 224:45 +CPU= 18.609
20:14:39 STEP 900 H= 225:00 +CPU= 24.001
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000900
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000900
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000900
20:14:57 STEP 901 H= 225:15 +CPU= 18.177
20:15:16 STEP 902 H= 225:30 +CPU= 18.799
20:15:34 STEP 903 H= 225:45 +CPU= 18.379
20:15:53 STEP 904 H= 226:00 +CPU= 19.185
20:16:12 STEP 905 H= 226:15 +CPU= 18.467
20:16:31 STEP 906 H= 226:30 +CPU= 18.559
20:16:49 STEP 907 H= 226:45 +CPU= 18.403
20:17:08 STEP 908 H= 227:00 +CPU= 19.248
20:17:27 STEP 909 H= 227:15 +CPU= 18.583
20:17:45 STEP 910 H= 227:30 +CPU= 18.282
20:18:04 STEP 911 H= 227:45 +CPU= 18.627
20:18:31 STEP 912 H= 228:00 +CPU= 26.434
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000912
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000912
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000912
20:18:49 STEP 913 H= 228:15 +CPU= 18.623
20:19:08 STEP 914 H= 228:30 +CPU= 18.902
20:19:27 STEP 915 H= 228:45 +CPU= 18.779
20:19:46 STEP 916 H= 229:00 +CPU= 19.257
20:20:05 STEP 917 H= 229:15 +CPU= 18.227
20:20:23 STEP 918 H= 229:30 +CPU= 18.638
20:23:15 STEP 919 H= 229:45 +CPU= 22.617
20:23:35 STEP 920 H= 230:00 +CPU= 19.157
20:23:53 STEP 921 H= 230:15 +CPU= 18.370
20:24:12 STEP 922 H= 230:30 +CPU= 18.571
20:24:31 STEP 923 H= 230:45 +CPU= 19.023
20:24:55 STEP 924 H= 231:00 +CPU= 23.910
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000924
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000924
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000924
20:25:13 STEP 925 H= 231:15 +CPU= 18.502
20:25:31 STEP 926 H= 231:30 +CPU= 18.168
20:25:50 STEP 927 H= 231:45 +CPU= 18.256
20:26:09 STEP 928 H= 232:00 +CPU= 18.922
20:26:27 STEP 929 H= 232:15 +CPU= 18.059
20:26:45 STEP 930 H= 232:30 +CPU= 18.628
20:27:04 STEP 931 H= 232:45 +CPU= 18.617
20:27:23 STEP 932 H= 233:00 +CPU= 19.208
20:27:42 STEP 933 H= 233:15 +CPU= 18.573
20:28:01 STEP 934 H= 233:30 +CPU= 18.835
20:28:20 STEP 935 H= 233:45 +CPU= 19.076
20:28:44 STEP 936 H= 234:00 +CPU= 24.285
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000936
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000936
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000936
20:29:02 STEP 937 H= 234:15 +CPU= 17.851
20:29:20 STEP 938 H= 234:30 +CPU= 18.216
20:29:39 STEP 939 H= 234:45 +CPU= 18.297
20:29:58 STEP 940 H= 235:00 +CPU= 19.107
20:30:16 STEP 941 H= 235:15 +CPU= 18.526
20:30:35 STEP 942 H= 235:30 +CPU= 18.913
20:30:54 STEP 943 H= 235:45 +CPU= 18.411
20:31:13 STEP 944 H= 236:00 +CPU= 19.044
20:31:31 STEP 945 H= 236:15 +CPU= 18.112
20:31:49 STEP 946 H= 236:30 +CPU= 18.340
20:32:08 STEP 947 H= 236:45 +CPU= 18.631
20:32:33 STEP 948 H= 237:00 +CPU= 24.897
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000948
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000948
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000948
20:32:52 STEP 949 H= 237:15 +CPU= 18.559
20:33:10 STEP 950 H= 237:30 +CPU= 18.378
20:33:28 STEP 951 H= 237:45 +CPU= 18.366
20:33:48 STEP 952 H= 238:00 +CPU= 19.378
20:34:07 STEP 953 H= 238:15 +CPU= 19.170
20:34:26 STEP 954 H= 238:30 +CPU= 18.632
20:34:44 STEP 955 H= 238:45 +CPU= 18.529
20:35:04 STEP 956 H= 239:00 +CPU= 19.245
20:35:22 STEP 957 H= 239:15 +CPU= 18.730
20:35:40 STEP 958 H= 239:30 +CPU= 18.074
20:35:59 STEP 959 H= 239:45 +CPU= 18.548
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01
DATE= 2016 10 1
NORMS AT NSTEP CNT4 960
SPECTRAL NORMS - LOG(PREHYDS) 0.115260902171495E+02
LEV VORTICITY DIVERGENCE TEMPERATURE KINETIC ENERGY
AVE 0.155592952582642E-04 0.266936313295805E-05 0.205870653560629E+03 0.136843734166399E+03
GPNORM HUMIDITY AVERAGE MINIMUM MAXIMUM
AVE 0.121333580168020E-02 0.100000000000000E-07 0.110737907526487E-01
GPNORM LIQUID WATER AVERAGE MINIMUM MAXIMUM
AVE 0.789471258073656E-06 0.000000000000000E+00 0.663950384866830E-03
GPNORM ICE WATER AVERAGE MINIMUM MAXIMUM
AVE 0.353154253990752E-06 0.000000000000000E+00 0.151235919049446E-03
GPNORM SNOW AVERAGE MINIMUM MAXIMUM
AVE 0.774932717347592E-06 0.000000000000000E+00 0.485183222421157E-03
GPNORM RAIN AVERAGE MINIMUM MAXIMUM
AVE 0.574641389233995E-07 0.000000000000000E+00 0.138387671355477E-03
GPNORM CLOUD FRACTION AVERAGE MINIMUM MAXIMUM
AVE 0.115786176539587E+00 0.000000000000000E+00 0.100000000000000E+01
GPNORM CLW AVERAGE MINIMUM MAXIMUM
AVE 0.787318562179431E-06 0.000000000000000E+00 0.655133849409147E-03
GPNORM CIW AVERAGE MINIMUM MAXIMUM
AVE 0.353003169564899E-06 0.000000000000000E+00 0.153168047531929E-03
GPNORM CC AVERAGE MINIMUM MAXIMUM
AVE 0.115679569025782E+00 0.000000000000000E+00 0.100000000000000E+01
GPNORM RFL AVERAGE MINIMUM MAXIMUM
AVE 0.260534278846523E-06 0.000000000000000E+00 0.660408312889587E-03
GPNORM SFL AVERAGE MINIMUM MAXIMUM
AVE 0.846495007886899E-06 0.000000000000000E+00 0.497522509835318E-03
GPNORM CRFL AVERAGE MINIMUM MAXIMUM
AVE 0.276268855146468E-07 0.000000000000000E+00 0.127609069224674E-03
GPNORM CSFL AVERAGE MINIMUM MAXIMUM
AVE 0.563676518181432E-07 0.000000000000000E+00 0.273658836367569E-03
GPNORM PFRC AVERAGE MINIMUM MAXIMUM
AVE 0.529766360461576E-01 0.000000000000000E+00 0.100000000000000E+01
NSTEP = 960 STEPO F00000000
CREATE RESTART FILE ./srf00100000.0001
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./srf00100000.0001 MODE=w
IO-STREAM CLOSED - ./srf00100000.0001
MPI-TASK: 1 - ************ BYTES IN 46 RECORDS TRANSFERRED IN 1.8085 SECONDS******** Mbytes/s, TOTAL TIME= 2.0660(87.5%)
CREATE RESTART CONTROL FILE ./rcf
WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00091200.0001
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 960 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+000960 MODE=w
IO-STREAM CLOSED - ICMGGh7zg+000960
MPI-TASK: 1 - 320081012 BYTES IN 6 RECORDS TRANSFERRED IN 0.0003 SECONDS******** Mbytes/s, TOTAL TIME= 0.0085( 3.5%)
NSTEP = 960 STEPO 0AA000000
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMSHh7zg+000960 MODE=w
IO-STREAM CLOSED - ./ICMSHh7zg+000960
MPI-TASK: 1 - 707399250 BYTES IN 202 RECORDS TRANSFERRED IN 0.0080 SECONDS 88424.9 Mbytes/s, TOTAL TIME= 0.2197( 3.6%)
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000960 MODE=w
IO-STREAM CLOSED - ./ICMUAh7zg+000960
MPI-TASK: 1 - 5121296192 BYTES IN 125 RECORDS TRANSFERRED IN 0.0052 SECONDS984864.7 Mbytes/s, TOTAL TIME= 0.1741( 3.0%)
STEPO_FPOS WAS : B00000F00E
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000960 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000960
MPI-TASK: 1 - 10 BYTES IN 2 RECORDS TRANSFERRED IN 0.0001 SECONDS 0.1 Mbytes/s, TOTAL TIME= 0.0034( 2.9%)
STEPO_FPOS WAS : V00000000Y
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000960 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000960
MPI-TASK: 1 - 311501283 BYTES IN 4 RECORDS TRANSFERRED IN 0.0002 SECONDS******** Mbytes/s, TOTAL TIME= 0.0051( 3.9%)
STEPO_FPOS WAS : T00000000M
NSTEP = 960 STEPO 0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat
20:35:40 0AAA00AAA STEPO 959 18.074 18.074 18.101 60:46 63:39 0.26493820941331E-05 2GB 0MB
20:35:59 F00000000 STEPO 960 18.696 18.696 18.735 61:05 63:58 0.26693631329581E-05 2GB 0MB
20:36:01 0AA000000 STEPO 960 2.206 2.206 2.218 61:07 64:00 0.26693631329581E-05 2GB 0MB
20:36:02 FULLPOS-B DYNFPOS 960 0.882 0.882 0.884 61:08 64:01 0.26693631329581E-05 2GB 0MB
20:36:05 FULLPOS-V DYNFPOS 960 2.702 2.702 2.707 61:10 64:04 0.26693631329581E-05 2GB 0MB
20:36:06 FULLPOS-T DYNFPOS 960 0.650 0.650 0.651 61:11 64:04 0.26693631329581E-05 2GB 0MB
20:36:06 0AAA00AAA STEPO 960 0.633 0.633 0.634 61:12 64:05 0.26693631329581E-05 2GB 0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf
CSTEP = " 960",
CTIME = "00100000 ",
NSTEPLPP = 201, 202, 49, 228226, 228227, 2*0, 948, 2*0,
IPRGPNSRES = 1,
IPRGPEWRES = 1,
IPRTRWRES = 1,
IPRTRVRES = 1,
GMASS0 = 0.000000000000000E+000,
GMASSI = 0.000000000000000E+000
/
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc/slots/4/progress_file_12292912.xml
<running_values>
<last_cpu_time>18329.450000</last_cpu_time>
<upload_file_number>9</upload_file_number>
<last_iter>960</last_iter>
<last_upload>778500</last_upload>
<model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3jd
(argv4) batchid: 1018
(argv5) wuid: 12292912
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/4
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/4/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/4/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_651068079fac398a28c088784343c8e0 to: /var/lib/boinc/slots/4/oifs_43r3_bl_a3jd_2016092300_20_1018_12292912.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/4/oifs_43r3_bl_a3jd_2016092300_20_1018_12292912.zip
ic_ancil_file: ic_ancil_12292912
ifsdata_file: ifsdata_12292912
climate_data_file: clim_data_12292912
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/4/ic_ancil_12292912.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/4/ic_ancil_12292912.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d57a568d0c430b89a90ffab3b11cb8d6 to: /var/lib/boinc/slots/4/ifsdata/ifsdata_12292912.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/4/ifsdata/ifsdata_12292912.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/4/159l_2/clim_data_12292912.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/4/159l_2/clim_data_12292912.zip
Checking for progress XML file: /var/lib/boinc/slots/4/progress_file_12292912.xml
Opened progress file ok : /var/lib/boinc/slots/4/progress_file_12292912.xml
-- Model is restarting --
Adjusting last_iter, 960, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/4/progress_file_12292912.xml
last_cpu_time: 18329
upload_file_number: 9
last_iter: 961
last_upload: 778500
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3jd_2016092300_20_1018_12292912_0_r1266141355
The child process has been launched with process id: 552162
Executing the command: /var/lib/boinc/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:552162:552162] [20240613:203713:1718303833:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000948
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000948
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000948
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3jd_2016092300_20_1018_12292912_0_r1266141355_9.zip
Uploading the intermediate file: upload_file_9.zip
20:37:35 STEP 960 H= 240:00 +CPU= 19.709
20:37:54 STEP 961 H= 240:15 +CPU= 18.463
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000960
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000960
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000960
20:38:12 STEP 962 H= 240:30 +CPU= 18.551
20:38:31 STEP 963 H= 240:45 +CPU= 18.486
20:38:50 STEP 964 H= 241:00 +CPU= 19.543
20:39:09 STEP 965 H= 241:15 +CPU= 18.454
20:39:27 STEP 966 H= 241:30 +CPU= 18.275
20:39:45 STEP 967 H= 241:45 +CPU= 18.086
20:40:05 STEP 968 H= 242:00 +CPU= 19.592
20:40:24 STEP 969 H= 242:15 +CPU= 18.964
20:40:43 STEP 970 H= 242:30 +CPU= 18.601
20:41:01 STEP 971 H= 242:45 +CPU= 18.405
20:41:26 STEP 972 H= 243:00 +CPU= 24.823
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000972
20:41:48 STEP 973 H= 243:15 +CPU= 19.301
20:42:07 STEP 974 H= 243:30 +CPU= 19.079
20:42:26 STEP 975 H= 243:45 +CPU= 19.079
20:42:46 STEP 976 H= 244:00 +CPU= 19.810
20:43:05 STEP 977 H= 244:15 +CPU= 18.707
20:43:24 STEP 978 H= 244:30 +CPU= 18.440
20:43:42 STEP 979 H= 244:45 +CPU= 18.088
20:44:01 STEP 980 H= 245:00 +CPU= 19.679
20:44:21 STEP 981 H= 245:15 +CPU= 19.155
20:44:39 STEP 982 H= 245:30 +CPU= 18.844
20:44:58 STEP 983 H= 245:45 +CPU= 18.368
20:45:22 STEP 984 H= 246:00 +CPU= 24.156
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000984
20:45:41 STEP 985 H= 246:15 +CPU= 19.141
20:46:00 STEP 986 H= 246:30 +CPU= 19.009
20:46:18 STEP 987 H= 246:45 +CPU= 18.034
20:46:37 STEP 988 H= 247:00 +CPU= 18.866
20:46:56 STEP 989 H= 247:15 +CPU= 18.225
20:47:14 STEP 990 H= 247:30 +CPU= 18.220
20:47:33 STEP 991 H= 247:45 +CPU= 18.747
20:47:52 STEP 992 H= 248:00 +CPU= 19.312
20:48:10 STEP 993 H= 248:15 +CPU= 18.361
20:48:29 STEP 994 H= 248:30 +CPU= 18.202
20:48:48 STEP 995 H= 248:45 +CPU= 19.219
20:49:12 STEP 996 H= 249:00 +CPU= 24.432
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000996
20:49:31 STEP 997 H= 249:15 +CPU= 18.522
20:49:49 STEP 998 H= 249:30 +CPU= 18.153
20:50:08 STEP 999 H= 249:45 +CPU= 19.059
20:50:27 STEP 1000 H= 250:00 +CPU= 19.301
20:50:46 STEP 1001 H= 250:15 +CPU= 18.642
20:51:04 STEP 1002 H= 250:30 +CPU= 18.252
20:51:23 STEP 1003 H= 250:45 +CPU= 18.699
20:51:43 STEP 1004 H= 251:00 +CPU= 19.421
20:52:01 STEP 1005 H= 251:15 +CPU= 18.349
20:52:20 STEP 1006 H= 251:30 +CPU= 18.446
20:52:38 STEP 1007 H= 251:45 +CPU= 18.313
20:53:04 STEP 1008 H= 252:00 +CPU= 26.227
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001008
20:53:23 STEP 1009 H= 252:15 +CPU= 18.759
20:53:42 STEP 1010 H= 252:30 +CPU= 18.888
20:54:00 STEP 1011 H= 252:45 +CPU= 18.072
20:54:19 STEP 1012 H= 253:00 +CPU= 18.808
20:54:38 STEP 1013 H= 253:15 +CPU= 18.659
20:54:56 STEP 1014 H= 253:30 +CPU= 18.519
20:55:15 STEP 1015 H= 253:45 +CPU= 18.506
20:55:34 STEP 1016 H= 254:00 +CPU= 19.254
20:55:52 STEP 1017 H= 254:15 +CPU= 18.062
20:56:11 STEP 1018 H= 254:30 +CPU= 18.561
20:56:29 STEP 1019 H= 254:45 +CPU= 18.493
20:56:53 STEP 1020 H= 255:00 +CPU= 24.234
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001020
20:57:12 STEP 1021 H= 255:15 +CPU= 18.240
20:57:30 STEP 1022 H= 255:30 +CPU= 18.085
20:57:48 STEP 1023 H= 255:45 +CPU= 18.371
20:58:07 STEP 1024 H= 256:00 +CPU= 19.205
20:58:26 STEP 1025 H= 256:15 +CPU= 18.713
20:58:45 STEP 1026 H= 256:30 +CPU= 18.515
20:59:03 STEP 1027 H= 256:45 +CPU= 18.680
20:59:23 STEP 1028 H= 257:00 +CPU= 19.235
20:59:41 STEP 1029 H= 257:15 +CPU= 18.623
21:00:00 STEP 1030 H= 257:30 +CPU= 18.178
21:00:18 STEP 1031 H= 257:45 +CPU= 18.094
21:00:42 STEP 1032 H= 258:00 +CPU= 24.558
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001032
21:01:01 STEP 1033 H= 258:15 +CPU= 18.341
21:01:19 STEP 1034 H= 258:30 +CPU= 18.673
21:01:38 STEP 1035 H= 258:45 +CPU= 18.416
21:01:57 STEP 1036 H= 259:00 +CPU= 18.871
21:02:15 STEP 1037 H= 259:15 +CPU= 18.354
21:02:33 STEP 1038 H= 259:30 +CPU= 18.262
21:02:52 STEP 1039 H= 259:45 +CPU= 18.479
21:03:11 STEP 1040 H= 260:00 +CPU= 19.081
21:03:30 STEP 1041 H= 260:15 +CPU= 18.504
21:03:49 STEP 1042 H= 260:30 +CPU= 18.836
21:04:07 STEP 1043 H= 260:45 +CPU= 18.866
21:04:32 STEP 1044 H= 261:00 +CPU= 24.508
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001044
21:04:51 STEP 1045 H= 261:15 +CPU= 18.748
21:05:09 STEP 1046 H= 261:30 +CPU= 18.570
21:05:28 STEP 1047 H= 261:45 +CPU= 18.812
21:05:48 STEP 1048 H= 262:00 +CPU= 19.418
21:06:06 STEP 1049 H= 262:15 +CPU= 18.358
21:06:24 STEP 1050 H= 262:30 +CPU= 18.323
21:06:43 STEP 1051 H= 262:45 +CPU= 18.507
21:07:02 STEP 1052 H= 263:00 +CPU= 19.045
21:07:21 STEP 1053 H= 263:15 +CPU= 18.806
21:07:40 STEP 1054 H= 263:30 +CPU= 19.015
21:07:59 STEP 1055 H= 263:45 +CPU= 18.742
21:08:26 STEP 1056 H= 264:00 +CPU= 26.742
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001056
21:08:44 STEP 1057 H= 264:15 +CPU= 18.765
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292912/ICMUAh7zg+001056
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3jd_2016092300_20_1018_12292912_0_r1266141355_10.zip
Uploading the intermediate file: upload_file_10.zip
21:09:03 STEP 1058 H= 264:30 +CPU= 18.700
21:09:22 STEP 1059 H= 264:45 +CPU= 18.630
21:09:42 STEP 1060 H= 265:00 +CPU= 19.707
21:10:01 STEP 1061 H= 265:15 +CPU= 19.035
21:10:20 STEP 1062 H= 265:30 +CPU= 19.098
21:10:39 STEP 1063 H= 265:45 +CPU= 18.906
21:10:58 STEP 1064 H= 266:00 +CPU= 19.185
21:11:17 STEP 1065 H= 266:15 +CPU= 19.034
21:11:35 STEP 1066 H= 266:30 +CPU= 18.331
21:11:54 STEP 1067 H= 266:45 +CPU= 18.670
21:12:19 STEP 1068 H= 267:00 +CPU= 24.797
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001068
21:12:38 STEP 1069 H= 267:15 +CPU= 19.023
21:12:57 STEP 1070 H= 267:30 +CPU= 18.648
21:13:15 STEP 1071 H= 267:45 +CPU= 18.351
21:13:35 STEP 1072 H= 268:00 +CPU= 19.517
21:13:54 STEP 1073 H= 268:15 +CPU= 19.007
21:14:13 STEP 1074 H= 268:30 +CPU= 18.729
21:14:31 STEP 1075 H= 268:45 +CPU= 18.354
21:14:50 STEP 1076 H= 269:00 +CPU= 18.730
21:15:09 STEP 1077 H= 269:15 +CPU= 18.860
21:15:27 STEP 1078 H= 269:30 +CPU= 18.582
21:15:46 STEP 1079 H= 269:45 +CPU= 18.678
21:16:10 STEP 1080 H= 270:00 +CPU= 24.402
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001080
21:16:29 STEP 1081 H= 270:15 +CPU= 18.382
21:16:48 STEP 1082 H= 270:30 +CPU= 18.665
21:17:06 STEP 1083 H= 270:45 +CPU= 18.411
21:17:25 STEP 1084 H= 271:00 +CPU= 19.229
21:17:44 STEP 1085 H= 271:15 +CPU= 18.468
21:18:03 STEP 1086 H= 271:30 +CPU= 18.724
21:18:21 STEP 1087 H= 271:45 +CPU= 18.338
21:18:40 STEP 1088 H= 272:00 +CPU= 19.430
21:18:59 STEP 1089 H= 272:15 +CPU= 18.729
21:19:18 STEP 1090 H= 272:30 +CPU= 18.877
21:19:37 STEP 1091 H= 272:45 +CPU= 18.419
21:20:01 STEP 1092 H= 273:00 +CPU= 24.849
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001092
21:20:21 STEP 1093 H= 273:15 +CPU= 19.171
21:20:39 STEP 1094 H= 273:30 +CPU= 18.696
21:20:58 STEP 1095 H= 273:45 +CPU= 18.287
21:21:17 STEP 1096 H= 274:00 +CPU= 19.471
21:24:26 STEP 1097 H= 274:15 +CPU= 23.864
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01
AVE 0.594770360851685E-07 0.000000000000000E+00 0.103673841419391E-03
GPNORM CLOUD FRACTION AVERAGE MINIMUM MAXIMUM
AVE 0.119119749347886E+00 0.000000000000000E+00 0.100000000000000E+01
GPNORM CLW AVERAGE MINIMUM MAXIMUM
AVE 0.104368716505753E-05 0.000000000000000E+00 0.688926411908735E-03
GPNORM CIW AVERAGE MINIMUM MAXIMUM
AVE 0.379408127823072E-06 0.000000000000000E+00 0.132116019536749E-03
GPNORM CC AVERAGE MINIMUM MAXIMUM
AVE 0.119175342306784E+00 0.000000000000000E+00 0.100000000000000E+01
GPNORM RFL AVERAGE MINIMUM MAXIMUM
AVE 0.270241374410246E-06 0.000000000000000E+00 0.461487800236031E-03
GPNORM SFL AVERAGE MINIMUM MAXIMUM
AVE 0.844833244914441E-06 0.000000000000000E+00 0.373774214812973E-03
GPNORM CRFL AVERAGE MINIMUM MAXIMUM
AVE 0.609156075604075E-07 0.000000000000000E+00 0.221319661791521E-03
GPNORM CSFL AVERAGE MINIMUM MAXIMUM
AVE 0.982021646370690E-07 0.000000000000000E+00 0.368346784783548E-03
GPNORM PFRC AVERAGE MINIMUM MAXIMUM
AVE 0.580732415182362E-01 0.000000000000000E+00 0.100000000000000E+01
NSTEP = 1092 STEPO A00000000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 1092 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+001092 MODE=w
IO-STREAM CLOSED - ICMGGh7zg+001092
MPI-TASK: 1 - 306100948 BYTES IN 6 RECORDS TRANSFERRED IN 0.0003 SECONDS******** Mbytes/s, TOTAL TIME= 0.0085( 3.5%)
NSTEP = 1092 STEPO 0AA000000
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMSHh7zg+001092 MODE=w
IO-STREAM CLOSED - ./ICMSHh7zg+001092
MPI-TASK: 1 - 707399250 BYTES IN 202 RECORDS TRANSFERRED IN 0.0073 SECONDS 96904.0 Mbytes/s, TOTAL TIME= 0.2217( 3.3%)
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+001092 MODE=w
IO-STREAM CLOSED - ./ICMUAh7zg+001092
MPI-TASK: 1 - 4897615168 BYTES IN 125 RECORDS TRANSFERRED IN 0.0046 SECONDS******** Mbytes/s, TOTAL TIME= 0.1698( 2.7%)
STEPO_FPOS WAS : B00000F00E
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+001092 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+001092
MPI-TASK: 1 - 10 BYTES IN 2 RECORDS TRANSFERRED IN 0.0001 SECONDS 0.1 Mbytes/s, TOTAL TIME= 0.0032( 3.1%)
STEPO_FPOS WAS : V00000000Y
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+001092 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+001092
MPI-TASK: 1 - 297521219 BYTES IN 4 RECORDS TRANSFERRED IN 0.0002 SECONDS******** Mbytes/s, TOTAL TIME= 0.0051( 3.9%)
STEPO_FPOS WAS : T00000000M
NSTEP = 1092 STEPO 0AAA00AAA
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 1092 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+001092 MODE=a
IO-STREAM CLOSED - ICMGGh7zg+001092
MPI-TASK: 1 - 306100948 BYTES IN 7 RECORDS TRANSFERRED IN 0.0005 SECONDS612201.9 Mbytes/s, TOTAL TIME= 0.0087( 5.7%)
21:20:01 STEP 1092 H= 273:00 +CPU= 24.849
NSTEP = 1093 STEPO 0AAA00AAA
21:20:21 STEP 1093 H= 273:15 +CPU= 19.171
NSTEP = 1094 STEPO 0AAA00AAA
21:20:39 STEP 1094 H= 273:30 +CPU= 18.696
ISTEP= 1095 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP= 1095 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
NSTEP = 1095 STEPO 0AAA00AAA
21:20:58 STEP 1095 H= 273:45 +CPU= 18.287
NSTEP = 1096 STEPO 0AAA00AAA
21:21:17 STEP 1096 H= 274:00 +CPU= 19.471
NSTEP = 1097 STEPO 0AAA00AAA
21:24:26 STEP 1097 H= 274:15 +CPU= 23.864
NSTEP = 1098 STEPO 0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat
21:19:42 0AAA00AAA STEPO 1092 0.630 0.630 0.631 42:20 42:28 0.28010123149390E-05 2GB 0MB
21:20:01 0AAA00AAA STEPO 1093 19.784 19.784 19.827 42:40 42:48 0.28010123149390E-05 2GB 0MB
21:20:21 0AAA00AAA STEPO 1094 19.171 19.171 19.211 42:59 43:07 0.28010123149390E-05 2GB 0MB
21:20:39 0AAA00AAA STEPO 1095 18.696 18.696 18.729 43:18 43:26 0.28010123149390E-05 2GB 0MB
21:20:58 0AAA00AAA STEPO 1096 18.287 18.287 18.320 43:36 43:44 0.28010123149390E-05 2GB 0MB
21:21:17 0AAA00AAA STEPO 1097 19.471 19.471 19.507 43:55 44:04 0.28010123149390E-05 2GB 0MB
21:24:26 0AAA00AAA STEPO 1098 23.865 23.865 188.752 44:19 47:13 0.28010123149390E-05 2GB 0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf
CSTEP = " 1056",
CTIME = "00110000 ",
NSTEPLPP = 201, 202, 49, 228226, 228227, 2*0, 1044, 2*0,
IPRGPNSRES = 1,
IPRGPEWRES = 1,
IPRTRWRES = 1,
IPRTRVRES = 1,
GMASS0 = 0.000000000000000E+000,
GMASSI = 0.000000000000000E+000
/
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc/slots/4/progress_file_12292912.xml
<running_values>
<last_cpu_time>20997.230000</last_cpu_time>
<upload_file_number>11</upload_file_number>
<last_iter>1098</last_iter>
<last_upload>951300</last_upload>
<model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
</stderr_txt>
]]>
