climateprediction.net home page
Task 22297929

Task 22297929

Name oifs_43r3_ps_0065_1987050100_123_956_12172709_2
Workunit 12172709
Created 14 Jan 2023, 19:55:14 UTC
Sent 14 Jan 2023, 20:34:29 UTC
Report deadline 13 Feb 2023, 20:34:29 UTC
Received 15 Jan 2023, 19:53:16 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1478928
Run time 10 hours 54 min 43 sec
CPU time 10 hours 54 min 37 sec
Validate state Invalid
Credit 10,644.48
Device peak FLOPS 8.02 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,769.08 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,415.10 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
   Line        Source             
oifs_43r3_model.e  0000000001CD9519  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D45366  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000873EE8  wavemdl_                  863  wavemdl.F
oifs_43r3_model.e  00000000007E0165  wvcouple_                 260  wvcouple.F90
oifs_43r3_model.e  000000000055E4AD  cnt4_                    1283  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0065
(argv4) batchid: 956
(argv5) wuid: 12172709
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a7a2b6e625194eb1268ac4881363053b to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0065_1987050100_123_956_12172709.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0065_1987050100_123_956_12172709.zip
ic_ancil_file: ic_ancil_12172709
ifsdata_file: ifsdata_12172709
climate_data_file: clim_data_12172709
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_be9b70fa59c61bdac2197dadfa93e058 to: /var/lib/boinc-client/slots/1/ic_ancil_12172709.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12172709.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1acf25cb695149c12517d9480648d0ef to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12172709.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12172709.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12172709.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12172709.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12172709.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12172709.xml
-- Model is restarting --
Adjusting last_iter, 1890, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12172709.xml
last_cpu_time: 31521
upload_file_number: 78
last_iter: 1871
last_upload: 6739200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102
The child process has been launched with process id: 2141
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:2141:2141] [20230115:123816:1673800696:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:38:41 STEP 1872 H=1872:00 +CPU= 21.877
  12:39:00 STEP 1873 H=1873:00 +CPU= 11.631
  12:39:11 STEP 1874 H=1874:00 +CPU= 11.413
  12:39:30 STEP 1875 H=1875:00 +CPU= 18.686
  12:39:43 STEP 1876 H=1876:00 +CPU= 12.208
  12:39:55 STEP 1877 H=1877:00 +CPU= 12.204
  12:40:14 STEP 1878 H=1878:00 +CPU= 19.360
  12:40:28 STEP 1879 H=1879:00 +CPU= 13.440
  12:40:42 STEP 1880 H=1880:00 +CPU= 14.019
  12:41:01 STEP 1881 H=1881:00 +CPU= 18.979
  12:41:13 STEP 1882 H=1882:00 +CPU= 12.431
  12:41:26 STEP 1883 H=1883:00 +CPU= 12.313
  12:41:49 STEP 1884 H=1884:00 +CPU= 23.051
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001884
  12:42:04 STEP 1885 H=1885:00 +CPU= 14.706
  12:42:17 STEP 1886 H=1886:00 +CPU= 13.048
  12:42:36 STEP 1887 H=1887:00 +CPU= 18.554
  12:42:49 STEP 1888 H=1888:00 +CPU= 12.817
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:43:18 STEP 1889 H=1889:00 +CPU= 14.993
  12:43:45 STEP 1890 H=1890:00 +CPU= 24.454
  12:44:00 STEP 1891 H=1891:00 +CPU= 14.583
  12:44:15 STEP 1892 H=1892:00 +CPU= 13.562
  12:44:35 STEP 1893 H=1893:00 +CPU= 19.951
  12:44:49 STEP 1894 H=1894:00 +CPU= 14.017
  12:45:05 STEP 1895 H=1895:00 +CPU= 14.807
  12:45:31 STEP 1896 H=1896:00 +CPU= 24.883
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001896
  12:45:43 STEP 1897 H=1897:00 +CPU= 12.391
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_78.zip
Uploading the intermediate file: upload_file_78.zip
  12:46:01 STEP 1898 H=1898:00 +CPU= 15.463
  12:46:26 STEP 1899 H=1899:00 +CPU= 23.804
  12:46:42 STEP 1900 H=1900:00 +CPU= 15.342
  12:46:58 STEP 1901 H=1901:00 +CPU= 15.222
  12:47:22 STEP 1902 H=1902:00 +CPU= 23.094
  12:47:37 STEP 1903 H=1903:00 +CPU= 14.925
  12:47:54 STEP 1904 H=1904:00 +CPU= 16.153
  12:48:17 STEP 1905 H=1905:00 +CPU= 22.906
  12:48:33 STEP 1906 H=1906:00 +CPU= 15.144
  12:48:48 STEP 1907 H=1907:00 +CPU= 14.665
  12:49:15 STEP 1908 H=1908:00 +CPU= 25.756
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001908
  12:49:28 STEP 1909 H=1909:00 +CPU= 13.341
  12:49:45 STEP 1910 H=1910:00 +CPU= 13.712
  12:50:09 STEP 1911 H=1911:00 +CPU= 22.112
  12:50:24 STEP 1912 H=1912:00 +CPU= 14.160
  12:50:39 STEP 1913 H=1913:00 +CPU= 14.220
  12:51:02 STEP 1914 H=1914:00 +CPU= 21.371
  12:51:17 STEP 1915 H=1915:00 +CPU= 15.038
  12:51:33 STEP 1916 H=1916:00 +CPU= 14.832
  12:51:57 STEP 1917 H=1917:00 +CPU= 22.386
  12:52:13 STEP 1918 H=1918:00 +CPU= 15.998
  12:52:29 STEP 1919 H=1919:00 +CPU= 14.551
  12:52:56 STEP 1920 H=1920:00 +CPU= 25.551
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_79.zip
Uploading the intermediate file: upload_file_79.zip
  12:53:12 STEP 1921 H=1921:00 +CPU= 15.433
  12:53:29 STEP 1922 H=1922:00 +CPU= 15.525
  12:53:53 STEP 1923 H=1923:00 +CPU= 22.924
  12:54:07 STEP 1924 H=1924:00 +CPU= 13.741
  12:54:22 STEP 1925 H=1925:00 +CPU= 14.036
  12:54:45 STEP 1926 H=1926:00 +CPU= 22.310
  12:55:00 STEP 1927 H=1927:00 +CPU= 14.919
  12:55:18 STEP 1928 H=1928:00 +CPU= 16.695
  12:55:43 STEP 1929 H=1929:00 +CPU= 24.391
  12:56:01 STEP 1930 H=1930:00 +CPU= 16.829
  12:56:17 STEP 1931 H=1931:00 +CPU= 15.517
  12:56:40 STEP 1932 H=1932:00 +CPU= 22.879
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001932
  12:56:55 STEP 1933 H=1933:00 +CPU= 14.350
  12:57:08 STEP 1934 H=1934:00 +CPU= 13.579
  12:57:32 STEP 1935 H=1935:00 +CPU= 22.435
  12:57:49 STEP 1936 H=1936:00 +CPU= 16.099
  12:58:05 STEP 1937 H=1937:00 +CPU= 15.192
  12:58:30 STEP 1938 H=1938:00 +CPU= 23.158
  12:58:45 STEP 1939 H=1939:00 +CPU= 14.342
  12:59:03 STEP 1940 H=1940:00 +CPU= 16.473
  12:59:28 STEP 1941 H=1941:00 +CPU= 23.621
  12:59:44 STEP 1942 H=1942:00 +CPU= 15.136
  13:00:00 STEP 1943 H=1943:00 +CPU= 14.946
  13:00:28 STEP 1944 H=1944:00 +CPU= 25.983
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_80.zip
Uploading the intermediate file: upload_file_80.zip
  13:00:51 STEP 1945 H=1945:00 +CPU= 16.234
  13:01:08 STEP 1946 H=1946:00 +CPU= 16.197
  13:01:31 STEP 1947 H=1947:00 +CPU= 22.266
  13:01:47 STEP 1948 H=1948:00 +CPU= 15.009
  13:02:03 STEP 1949 H=1949:00 +CPU= 14.387
  13:02:28 STEP 1950 H=1950:00 +CPU= 23.838
  13:02:44 STEP 1951 H=1951:00 +CPU= 14.868
  13:03:00 STEP 1952 H=1952:00 +CPU= 15.412
  13:03:26 STEP 1953 H=1953:00 +CPU= 23.867
  13:03:42 STEP 1954 H=1954:00 +CPU= 15.351
  13:04:00 STEP 1955 H=1955:00 +CPU= 16.292
  13:04:27 STEP 1956 H=1956:00 +CPU= 25.370
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001956
  13:04:44 STEP 1957 H=1957:00 +CPU= 16.190
  13:05:01 STEP 1958 H=1958:00 +CPU= 16.131
  13:05:27 STEP 1959 H=1959:00 +CPU= 23.742
  13:05:43 STEP 1960 H=1960:00 +CPU= 15.119
  13:05:58 STEP 1961 H=1961:00 +CPU= 14.304
  13:06:23 STEP 1962 H=1962:00 +CPU= 23.549
  13:06:39 STEP 1963 H=1963:00 +CPU= 14.699
  13:06:56 STEP 1964 H=1964:00 +CPU= 16.170
  13:07:18 STEP 1965 H=1965:00 +CPU= 21.651
  13:07:31 STEP 1966 H=1966:00 +CPU= 13.369
  13:07:45 STEP 1967 H=1967:00 +CPU= 13.348
  13:08:10 STEP 1968 H=1968:00 +CPU= 24.668
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_81.zip
Uploading the intermediate file: upload_file_81.zip
  13:08:23 STEP 1969 H=1969:00 +CPU= 13.596
  13:08:38 STEP 1970 H=1970:00 +CPU= 14.688
  13:09:01 STEP 1971 H=1971:00 +CPU= 22.642
  13:09:16 STEP 1972 H=1972:00 +CPU= 14.682
  13:09:29 STEP 1973 H=1973:00 +CPU= 13.440
  13:09:50 STEP 1974 H=1974:00 +CPU= 21.085
  13:10:04 STEP 1975 H=1975:00 +CPU= 13.440
  13:10:18 STEP 1976 H=1976:00 +CPU= 14.490
  13:10:42 STEP 1977 H=1977:00 +CPU= 23.272
  13:10:59 STEP 1978 H=1978:00 +CPU= 16.691
  13:11:14 STEP 1979 H=1979:00 +CPU= 14.778
  13:11:37 STEP 1980 H=1980:00 +CPU= 22.668
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001980
  13:11:51 STEP 1981 H=1981:00 +CPU= 13.704
  13:12:04 STEP 1982 H=1982:00 +CPU= 13.852
  13:12:29 STEP 1983 H=1983:00 +CPU= 23.140
  13:12:45 STEP 1984 H=1984:00 +CPU= 15.217
  13:13:01 STEP 1985 H=1985:00 +CPU= 14.400
  13:13:26 STEP 1986 H=1986:00 +CPU= 23.822
  13:13:42 STEP 1987 H=1987:00 +CPU= 15.403
  13:13:58 STEP 1988 H=1988:00 +CPU= 14.690
  13:14:22 STEP 1989 H=1989:00 +CPU= 22.004
  13:14:37 STEP 1990 H=1990:00 +CPU= 15.007
  13:14:55 STEP 1991 H=1991:00 +CPU= 16.349
  13:15:31 STEP 1992 H=1992:00 +CPU= 28.521
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_82.zip
Uploading the intermediate file: upload_file_82.zip
  13:15:46 STEP 1993 H=1993:00 +CPU= 14.476
  13:16:01 STEP 1994 H=1994:00 +CPU= 14.401
  13:16:25 STEP 1995 H=1995:00 +CPU= 22.665
  13:16:42 STEP 1996 H=1996:00 +CPU= 15.908
  13:17:04 STEP 1997 H=1997:00 +CPU= 16.070
  13:17:29 STEP 1998 H=1998:00 +CPU= 23.417
  13:17:45 STEP 1999 H=1999:00 +CPU= 14.733
  13:18:01 STEP 2000 H=2000:00 +CPU= 14.967
  13:18:25 STEP 2001 H=2001:00 +CPU= 23.362
  13:18:43 STEP 2002 H=2002:00 +CPU= 16.273
  13:18:58 STEP 2003 H=2003:00 +CPU= 14.378
  13:19:24 STEP 2004 H=2004:00 +CPU= 24.440
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002004
  13:19:39 STEP 2005 H=2005:00 +CPU= 14.668
  13:19:56 STEP 2006 H=2006:00 +CPU= 15.558
  13:20:21 STEP 2007 H=2007:00 +CPU= 23.982
  13:20:37 STEP 2008 H=2008:00 +CPU= 14.981
  13:20:52 STEP 2009 H=2009:00 +CPU= 14.342
  13:21:16 STEP 2010 H=2010:00 +CPU= 22.370
  13:21:33 STEP 2011 H=2011:00 +CPU= 15.958
  13:21:50 STEP 2012 H=2012:00 +CPU= 16.185
  13:22:16 STEP 2013 H=2013:00 +CPU= 24.630
  13:22:33 STEP 2014 H=2014:00 +CPU= 15.162
  13:22:49 STEP 2015 H=2015:00 +CPU= 15.803
  13:23:19 STEP 2016 H=2016:00 +CPU= 27.476
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_83.zip
Uploading the intermediate file: upload_file_83.zip
  13:23:37 STEP 2017 H=2017:00 +CPU= 17.000
  13:23:52 STEP 2018 H=2018:00 +CPU= 15.118
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:24:29 STEP 2019 H=2019:00 +CPU= 24.425
  13:24:45 STEP 2020 H=2020:00 +CPU= 15.734
  13:25:01 STEP 2021 H=2021:00 +CPU= 15.068
  13:25:27 STEP 2022 H=2022:00 +CPU= 24.866
  13:25:44 STEP 2023 H=2023:00 +CPU= 15.993
  13:26:01 STEP 2024 H=2024:00 +CPU= 15.921
  13:26:26 STEP 2025 H=2025:00 +CPU= 24.452
  13:26:46 STEP 2026 H=2026:00 +CPU= 17.997
  13:27:04 STEP 2027 H=2027:00 +CPU= 17.307
  13:27:36 STEP 2028 H=2028:00 +CPU= 29.798
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002028
  13:27:54 STEP 2029 H=2029:00 +CPU= 17.638
  13:28:09 STEP 2030 H=2030:00 +CPU= 14.639
  13:28:30 STEP 2031 H=2031:00 +CPU= 20.724
  13:28:44 STEP 2032 H=2032:00 +CPU= 13.446
  13:28:56 STEP 2033 H=2033:00 +CPU= 12.448
  13:29:18 STEP 2034 H=2034:00 +CPU= 21.954
  13:29:33 STEP 2035 H=2035:00 +CPU= 14.917
  13:29:49 STEP 2036 H=2036:00 +CPU= 15.559
  13:30:11 STEP 2037 H=2037:00 +CPU= 22.228
  13:30:25 STEP 2038 H=2038:00 +CPU= 13.458
  13:30:38 STEP 2039 H=2039:00 +CPU= 13.522
  13:31:03 STEP 2040 H=2040:00 +CPU= 24.571
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_84.zip
Uploading the intermediate file: upload_file_84.zip
  13:31:17 STEP 2041 H=2041:00 +CPU= 14.145
  13:31:32 STEP 2042 H=2042:00 +CPU= 14.984
  13:31:56 STEP 2043 H=2043:00 +CPU= 23.777
  13:32:10 STEP 2044 H=2044:00 +CPU= 13.925
  13:32:23 STEP 2045 H=2045:00 +CPU= 13.028
  13:32:45 STEP 2046 H=2046:00 +CPU= 21.393
  13:32:59 STEP 2047 H=2047:00 +CPU= 13.858
  13:33:14 STEP 2048 H=2048:00 +CPU= 14.639
  13:33:36 STEP 2049 H=2049:00 +CPU= 22.805
  13:33:50 STEP 2050 H=2050:00 +CPU= 13.885
  13:34:03 STEP 2051 H=2051:00 +CPU= 12.703
  13:34:27 STEP 2052 H=2052:00 +CPU= 23.512
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002052
  13:34:41 STEP 2053 H=2053:00 +CPU= 14.479
  13:34:55 STEP 2054 H=2054:00 +CPU= 13.965
  13:35:16 STEP 2055 H=2055:00 +CPU= 20.857
  13:35:30 STEP 2056 H=2056:00 +CPU= 13.543
  13:35:43 STEP 2057 H=2057:00 +CPU= 12.727
  13:36:05 STEP 2058 H=2058:00 +CPU= 22.605
  13:36:20 STEP 2059 H=2059:00 +CPU= 14.173
  13:36:33 STEP 2060 H=2060:00 +CPU= 12.996
  13:36:54 STEP 2061 H=2061:00 +CPU= 21.054
  13:37:08 STEP 2062 H=2062:00 +CPU= 13.652
  13:37:21 STEP 2063 H=2063:00 +CPU= 13.365
  13:37:54 STEP 2064 H=2064:00 +CPU= 23.337
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_85.zip
Uploading the intermediate file: upload_file_85.zip
  13:38:09 STEP 2065 H=2065:00 +CPU= 15.246
  13:38:24 STEP 2066 H=2066:00 +CPU= 14.581
  13:38:45 STEP 2067 H=2067:00 +CPU= 20.665
  13:38:58 STEP 2068 H=2068:00 +CPU= 13.643
  13:39:12 STEP 2069 H=2069:00 +CPU= 13.204
  13:39:34 STEP 2070 H=2070:00 +CPU= 22.299
  13:39:47 STEP 2071 H=2071:00 +CPU= 13.369
  13:40:01 STEP 2072 H=2072:00 +CPU= 13.566
  13:40:22 STEP 2073 H=2073:00 +CPU= 20.693
  13:40:36 STEP 2074 H=2074:00 +CPU= 14.116
  13:40:51 STEP 2075 H=2075:00 +CPU= 14.602
  13:41:17 STEP 2076 H=2076:00 +CPU= 25.664
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002076
  13:41:31 STEP 2077 H=2077:00 +CPU= 14.249
  13:41:44 STEP 2078 H=2078:00 +CPU= 13.321
  13:42:05 STEP 2079 H=2079:00 +CPU= 20.544
  13:42:18 STEP 2080 H=2080:00 +CPU= 13.493
  13:42:32 STEP 2081 H=2081:00 +CPU= 13.570
  13:42:55 STEP 2082 H=2082:00 +CPU= 22.333
  13:43:08 STEP 2083 H=2083:00 +CPU= 13.452
  13:43:22 STEP 2084 H=2084:00 +CPU= 13.601
  13:43:43 STEP 2085 H=2085:00 +CPU= 20.734
  13:43:57 STEP 2086 H=2086:00 +CPU= 14.459
  13:44:13 STEP 2087 H=2087:00 +CPU= 15.426
  13:44:38 STEP 2088 H=2088:00 +CPU= 24.622
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_86.zip
Uploading the intermediate file: upload_file_86.zip
  13:44:51 STEP 2089 H=2089:00 +CPU= 12.901
  13:45:04 STEP 2090 H=2090:00 +CPU= 13.300
  13:45:26 STEP 2091 H=2091:00 +CPU= 21.841
  13:45:44 STEP 2092 H=2092:00 +CPU= 15.215
  13:45:58 STEP 2093 H=2093:00 +CPU= 13.989
  13:46:20 STEP 2094 H=2094:00 +CPU= 21.581
  13:46:33 STEP 2095 H=2095:00 +CPU= 13.236
  13:46:46 STEP 2096 H=2096:00 +CPU= 13.317
  13:47:09 STEP 2097 H=2097:00 +CPU= 22.813
  13:47:23 STEP 2098 H=2098:00 +CPU= 14.124
  13:47:37 STEP 2099 H=2099:00 +CPU= 12.990
  13:48:00 STEP 2100 H=2100:00 +CPU= 23.238
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002100
  13:48:13 STEP 2101 H=2101:00 +CPU= 12.980
  13:48:27 STEP 2102 H=2102:00 +CPU= 14.456
  13:48:51 STEP 2103 H=2103:00 +CPU= 23.777
  13:49:07 STEP 2104 H=2104:00 +CPU= 15.416
  13:49:21 STEP 2105 H=2105:00 +CPU= 14.360
  13:49:43 STEP 2106 H=2106:00 +CPU= 21.344
  13:49:56 STEP 2107 H=2107:00 +CPU= 13.216
  13:50:09 STEP 2108 H=2108:00 +CPU= 13.070
  13:50:31 STEP 2109 H=2109:00 +CPU= 22.143
  13:50:47 STEP 2110 H=2110:00 +CPU= 15.376
  13:51:01 STEP 2111 H=2111:00 +CPU= 14.384
  13:51:30 STEP 2112 H=2112:00 +CPU= 27.410
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_87.zip
Uploading the intermediate file: upload_file_87.zip
  13:51:44 STEP 2113 H=2113:00 +CPU= 14.543
  13:51:57 STEP 2114 H=2114:00 +CPU= 12.517
Uploading trickle at timestep: 7603200
  13:52:18 STEP 2115 H=2115:00 +CPU= 21.401
  13:52:32 STEP 2116 H=2116:00 +CPU= 13.916
  13:52:45 STEP 2117 H=2117:00 +CPU= 12.619
  13:53:09 STEP 2118 H=2118:00 +CPU= 23.521
  13:53:23 STEP 2119 H=2119:00 +CPU= 14.316
  13:53:36 STEP 2120 H=2120:00 +CPU= 12.833
  13:53:57 STEP 2121 H=2121:00 +CPU= 21.147
  13:54:11 STEP 2122 H=2122:00 +CPU= 13.729
  13:54:24 STEP 2123 H=2123:00 +CPU= 13.153
  13:54:51 STEP 2124 H=2124:00 +CPU= 26.063
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002124
  13:55:06 STEP 2125 H=2125:00 +CPU= 14.958
  13:55:19 STEP 2126 H=2126:00 +CPU= 13.420
  13:55:41 STEP 2127 H=2127:00 +CPU= 21.233
  13:55:54 STEP 2128 H=2128:00 +CPU= 13.426
  13:56:07 STEP 2129 H=2129:00 +CPU= 12.552
  13:56:30 STEP 2130 H=2130:00 +CPU= 23.217
  13:56:44 STEP 2131 H=2131:00 +CPU= 13.549
  13:56:57 STEP 2132 H=2132:00 +CPU= 12.710
  13:57:18 STEP 2133 H=2133:00 +CPU= 21.190
  13:57:32 STEP 2134 H=2134:00 +CPU= 13.748
  13:57:45 STEP 2135 H=2135:00 +CPU= 12.973
  13:58:12 STEP 2136 H=2136:00 +CPU= 27.311
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_88.zip
Uploading the intermediate file: upload_file_88.zip
  13:58:27 STEP 2137 H=2137:00 +CPU= 15.091
  13:58:49 STEP 2138 H=2138:00 +CPU= 15.440
  13:59:12 STEP 2139 H=2139:00 +CPU= 22.855
  13:59:26 STEP 2140 H=2140:00 +CPU= 13.325
  13:59:39 STEP 2141 H=2141:00 +CPU= 13.253
  14:00:00 STEP 2142 H=2142:00 +CPU= 21.383
  14:00:14 STEP 2143 H=2143:00 +CPU= 13.557
  14:00:29 STEP 2144 H=2144:00 +CPU= 15.016
  14:00:53 STEP 2145 H=2145:00 +CPU= 23.820
  14:01:08 STEP 2146 H=2146:00 +CPU= 15.136
  14:01:23 STEP 2147 H=2147:00 +CPU= 14.679
  14:01:46 STEP 2148 H=2148:00 +CPU= 22.953
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002148
  14:01:59 STEP 2149 H=2149:00 +CPU= 13.006
  14:02:13 STEP 2150 H=2150:00 +CPU= 13.464
  14:02:35 STEP 2151 H=2151:00 +CPU= 21.978
  14:02:50 STEP 2152 H=2152:00 +CPU= 14.789
  14:03:05 STEP 2153 H=2153:00 +CPU= 14.879
  14:03:28 STEP 2154 H=2154:00 +CPU= 23.132
  14:03:41 STEP 2155 H=2155:00 +CPU= 12.824
  14:03:54 STEP 2156 H=2156:00 +CPU= 13.425
  14:04:16 STEP 2157 H=2157:00 +CPU= 21.536
  14:04:30 STEP 2158 H=2158:00 +CPU= 13.609
  14:04:45 STEP 2159 H=2159:00 +CPU= 15.538
  14:05:14 STEP 2160 H=2160:00 +CPU= 28.178
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_89.zip
Uploading the intermediate file: upload_file_89.zip
  14:05:29 STEP 2161 H=2161:00 +CPU= 14.523
  14:05:44 STEP 2162 H=2162:00 +CPU= 14.403
  14:06:05 STEP 2163 H=2163:00 +CPU= 20.978
  14:06:18 STEP 2164 H=2164:00 +CPU= 13.209
  14:06:31 STEP 2165 H=2165:00 +CPU= 13.401
  14:06:54 STEP 2166 H=2166:00 +CPU= 22.617
  14:07:09 STEP 2167 H=2167:00 +CPU= 14.848
  14:07:24 STEP 2168 H=2168:00 +CPU= 15.152
  14:07:47 STEP 2169 H=2169:00 +CPU= 22.784
  14:08:01 STEP 2170 H=2170:00 +CPU= 13.504
  14:08:14 STEP 2171 H=2171:00 +CPU= 13.399
  14:08:37 STEP 2172 H=2172:00 +CPU= 22.599
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002172
  14:08:50 STEP 2173 H=2173:00 +CPU= 13.055
  14:09:05 STEP 2174 H=2174:00 +CPU= 15.025
  14:09:29 STEP 2175 H=2175:00 +CPU= 23.339
  14:09:44 STEP 2176 H=2176:00 +CPU= 15.289
  14:09:59 STEP 2177 H=2177:00 +CPU= 14.689
  14:10:20 STEP 2178 H=2178:00 +CPU= 21.214
  14:10:34 STEP 2179 H=2179:00 +CPU= 13.360
  14:10:47 STEP 2180 H=2180:00 +CPU= 13.483
  14:11:08 STEP 2181 H=2181:00 +CPU= 20.851
  14:11:23 STEP 2182 H=2182:00 +CPU= 15.128
  14:11:39 STEP 2183 H=2183:00 +CPU= 15.228
  14:12:05 STEP 2184 H=2184:00 +CPU= 24.973
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_90.zip
Uploading the intermediate file: upload_file_90.zip
  14:12:18 STEP 2185 H=2185:00 +CPU= 13.252
  14:12:31 STEP 2186 H=2186:00 +CPU= 13.313
  14:12:53 STEP 2187 H=2187:00 +CPU= 21.375
  14:13:13 STEP 2188 H=2188:00 +CPU= 15.444
  14:13:28 STEP 2189 H=2189:00 +CPU= 14.557
  14:13:51 STEP 2190 H=2190:00 +CPU= 22.881
  14:14:04 STEP 2191 H=2191:00 +CPU= 13.380
  14:14:17 STEP 2192 H=2192:00 +CPU= 13.345
  14:14:38 STEP 2193 H=2193:00 +CPU= 20.765
  14:14:52 STEP 2194 H=2194:00 +CPU= 14.008
  14:15:07 STEP 2195 H=2195:00 +CPU= 14.779
  14:15:33 STEP 2196 H=2196:00 +CPU= 26.088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002196
  14:15:47 STEP 2197 H=2197:00 +CPU= 13.686
  14:16:01 STEP 2198 H=2198:00 +CPU= 13.315
  14:16:22 STEP 2199 H=2199:00 +CPU= 21.390
  14:16:36 STEP 2200 H=2200:00 +CPU= 13.572
  14:16:50 STEP 2201 H=2201:00 +CPU= 13.873
  14:17:13 STEP 2202 H=2202:00 +CPU= 22.725
  14:17:26 STEP 2203 H=2203:00 +CPU= 13.163
  14:17:39 STEP 2204 H=2204:00 +CPU= 13.143
  14:18:00 STEP 2205 H=2205:00 +CPU= 21.169
  14:18:15 STEP 2206 H=2206:00 +CPU= 14.390
  14:18:29 STEP 2207 H=2207:00 +CPU= 14.720
  14:18:59 STEP 2208 H=2208:00 +CPU= 27.470
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_91.zip
Uploading the intermediate file: upload_file_91.zip
  14:19:13 STEP 2209 H=2209:00 +CPU= 13.088
  14:19:26 STEP 2210 H=2210:00 +CPU= 12.987
  14:19:47 STEP 2211 H=2211:00 +CPU= 21.280
  14:20:00 STEP 2212 H=2212:00 +CPU= 13.252
  14:20:14 STEP 2213 H=2213:00 +CPU= 13.584
  14:20:37 STEP 2214 H=2214:00 +CPU= 22.379
  14:20:50 STEP 2215 H=2215:00 +CPU= 13.005
  14:21:03 STEP 2216 H=2216:00 +CPU= 12.943
  14:21:24 STEP 2217 H=2217:00 +CPU= 21.505
  14:21:39 STEP 2218 H=2218:00 +CPU= 14.376
  14:21:52 STEP 2219 H=2219:00 +CPU= 13.587
  14:22:16 STEP 2220 H=2220:00 +CPU= 23.390
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002220
  14:22:29 STEP 2221 H=2221:00 +CPU= 13.157
  14:22:42 STEP 2222 H=2222:00 +CPU= 12.953
  14:23:05 STEP 2223 H=2223:00 +CPU= 22.534
  14:23:20 STEP 2224 H=2224:00 +CPU= 14.644
  14:23:34 STEP 2225 H=2225:00 +CPU= 14.477
  14:23:57 STEP 2226 H=2226:00 +CPU= 22.915
  14:24:10 STEP 2227 H=2227:00 +CPU= 12.964
  14:24:23 STEP 2228 H=2228:00 +CPU= 12.969
  14:24:45 STEP 2229 H=2229:00 +CPU= 21.544
  14:25:00 STEP 2230 H=2230:00 +CPU= 14.589
  14:25:14 STEP 2231 H=2231:00 +CPU= 14.199
  14:25:39 STEP 2232 H=2232:00 +CPU= 24.882
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_92.zip
Uploading the intermediate file: upload_file_92.zip
  14:25:52 STEP 2233 H=2233:00 +CPU= 13.204
  14:26:06 STEP 2234 H=2234:00 +CPU= 13.569
  14:26:29 STEP 2235 H=2235:00 +CPU= 22.802
  14:26:45 STEP 2236 H=2236:00 +CPU= 14.646
  14:26:59 STEP 2237 H=2237:00 +CPU= 13.237
  14:27:20 STEP 2238 H=2238:00 +CPU= 21.553
  14:27:33 STEP 2239 H=2239:00 +CPU= 13.155
  14:27:47 STEP 2240 H=2240:00 +CPU= 13.787
  14:28:11 STEP 2241 H=2241:00 +CPU= 23.248
  14:28:25 STEP 2242 H=2242:00 +CPU= 14.708
  14:28:39 STEP 2243 H=2243:00 +CPU= 13.292
  14:29:02 STEP 2244 H=2244:00 +CPU= 23.012
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002244
  14:29:15 STEP 2245 H=2245:00 +CPU= 13.244
  14:29:30 STEP 2246 H=2246:00 +CPU= 14.806
  14:29:53 STEP 2247 H=2247:00 +CPU= 23.085
  14:30:09 STEP 2248 H=2248:00 +CPU= 15.541
  14:30:23 STEP 2249 H=2249:00 +CPU= 13.998
  14:30:44 STEP 2250 H=2250:00 +CPU= 21.383
  14:30:58 STEP 2251 H=2251:00 +CPU= 13.437
  14:31:12 STEP 2252 H=2252:00 +CPU= 13.501
  14:31:34 STEP 2253 H=2253:00 +CPU= 22.466
  14:31:49 STEP 2254 H=2254:00 +CPU= 14.448
  14:32:03 STEP 2255 H=2255:00 +CPU= 13.874
  14:32:26 STEP 2256 H=2256:00 +CPU= 22.777
  14:32:36 STEP 2257 H=2257:00 +CPU= 10.793
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_93.zip
Uploading the intermediate file: upload_file_93.zip
  14:32:49 STEP 2258 H=2258:00 +CPU= 12.850
  14:33:16 STEP 2259 H=2259:00 +CPU= 24.887
  14:33:33 STEP 2260 H=2260:00 +CPU= 16.737
  14:33:49 STEP 2261 H=2261:00 +CPU= 14.779
  14:34:16 STEP 2262 H=2262:00 +CPU= 25.418
  14:34:34 STEP 2263 H=2263:00 +CPU= 16.860
  14:34:50 STEP 2264 H=2264:00 +CPU= 15.488
  14:35:15 STEP 2265 H=2265:00 +CPU= 24.368
  14:35:34 STEP 2266 H=2266:00 +CPU= 17.412
  14:35:52 STEP 2267 H=2267:00 +CPU= 16.945
  14:36:21 STEP 2268 H=2268:00 +CPU= 27.987
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002268
  14:36:39 STEP 2269 H=2269:00 +CPU= 16.350
  14:36:55 STEP 2270 H=2270:00 +CPU= 15.266
  14:37:23 STEP 2271 H=2271:00 +CPU= 26.417
  14:37:41 STEP 2272 H=2272:00 +CPU= 17.199
  14:37:57 STEP 2273 H=2273:00 +CPU= 15.151
  14:38:23 STEP 2274 H=2274:00 +CPU= 24.906
  14:38:41 STEP 2275 H=2275:00 +CPU= 16.735
  14:38:59 STEP 2276 H=2276:00 +CPU= 17.310
  14:39:27 STEP 2277 H=2277:00 +CPU= 26.293
  14:39:44 STEP 2278 H=2278:00 +CPU= 15.998
  14:40:01 STEP 2279 H=2279:00 +CPU= 15.734
  14:40:32 STEP 2280 H=2280:00 +CPU= 29.272
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_94.zip
Uploading the intermediate file: upload_file_94.zip
  14:40:52 STEP 2281 H=2281:00 +CPU= 18.231
  14:41:11 STEP 2282 H=2282:00 +CPU= 17.889
  14:41:39 STEP 2283 H=2283:00 +CPU= 26.752
  14:41:56 STEP 2284 H=2284:00 +CPU= 16.010
  14:42:13 STEP 2285 H=2285:00 +CPU= 15.500
  14:42:39 STEP 2286 H=2286:00 +CPU= 24.879
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0065
(argv4) batchid: 956
(argv5) wuid: 12172709
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a7a2b6e625194eb1268ac4881363053b to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0065_1987050100_123_956_12172709.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0065_1987050100_123_956_12172709.zip
ic_ancil_file: ic_ancil_12172709
ifsdata_file: ifsdata_12172709
climate_data_file: clim_data_12172709
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_be9b70fa59c61bdac2197dadfa93e058 to: /var/lib/boinc-client/slots/1/ic_ancil_12172709.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12172709.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1acf25cb695149c12517d9480648d0ef to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12172709.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12172709.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12172709.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12172709.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12172709.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12172709.xml
-- Model is restarting --
Adjusting last_iter, 2287, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12172709.xml
last_cpu_time: 38721
upload_file_number: 95
last_iter: 2279
last_upload: 8208000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102
The child process has been launched with process id: 578786
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:saint-marie:1:1:578786:578786] [20230115:144353:1673808233:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:44:22 STEP 2280 H=2280:00 +CPU= 25.213
  14:44:38 STEP 2281 H=2281:00 +CPU= 14.706
  14:44:53 STEP 2282 H=2282:00 +CPU= 14.161
  14:45:16 STEP 2283 H=2283:00 +CPU= 21.830
  14:45:31 STEP 2284 H=2284:00 +CPU= 14.821
  14:45:48 STEP 2285 H=2285:00 +CPU= 15.528
  14:46:12 STEP 2286 H=2286:00 +CPU= 22.808
  14:46:27 STEP 2287 H=2287:00 +CPU= 14.373
  14:46:43 STEP 2288 H=2288:00 +CPU= 14.781
  14:47:08 STEP 2289 H=2289:00 +CPU= 23.310
  14:47:24 STEP 2290 H=2290:00 +CPU= 14.968
  14:47:39 STEP 2291 H=2291:00 +CPU= 14.415
  14:48:05 STEP 2292 H=2292:00 +CPU= 24.983
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002292
  14:48:22 STEP 2293 H=2293:00 +CPU= 15.759
  14:48:38 STEP 2294 H=2294:00 +CPU= 14.894
  14:49:02 STEP 2295 H=2295:00 +CPU= 22.475
  14:49:18 STEP 2296 H=2296:00 +CPU= 15.137
  14:49:34 STEP 2297 H=2297:00 +CPU= 15.537
  14:49:59 STEP 2298 H=2298:00 +CPU= 23.640
  14:50:15 STEP 2299 H=2299:00 +CPU= 14.618
  14:50:30 STEP 2300 H=2300:00 +CPU= 14.699
  14:50:52 STEP 2301 H=2301:00 +CPU= 20.987
  14:51:08 STEP 2302 H=2302:00 +CPU= 14.492
  14:51:26 STEP 2303 H=2303:00 +CPU= 14.501
  14:51:52 STEP 2304 H=2304:00 +CPU= 24.542
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172709/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0065_1987050100_123_956_12172709_2_r1138692102_95.zip
Uploading the intermediate file: upload_file_95.zip
  14:52:07 STEP 2305 H=2305:00 +CPU= 13.942
  14:52:22 STEP 2306 H=2306:00 +CPU= 14.292
  14:52:47 STEP 2307 H=2307:00 +CPU= 23.439
  14:53:02 STEP 2308 H=2308:00 +CPU= 14.790
  14:53:17 STEP 2309 H=2309:00 +CPU= 14.028
  14:53:39 STEP 2310 H=2310:00 +CPU= 20.919
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:21:08 1478928 22297929 oifs_43r3_ps_0065_1987050100_123_956_12172709_2 7,603,200 35,722 0.0047
09 May 2023 16:20:12 1478928 22297929 oifs_43r3_ps_0065_1987050100_123_956_12172709_2 6,048,000 27,710 0.0046
09 May 2023 16:20:12 1478928 22297929 oifs_43r3_ps_0065_1987050100_123_956_12172709_2 6,048,000 27,710 0.0046
09 May 2023 16:20:04 1478928 22297929 oifs_43r3_ps_0065_1987050100_123_956_12172709_2 4,579,200 20,692 0.0045
09 May 2023 16:20:04 1478928 22297929 oifs_43r3_ps_0065_1987050100_123_956_12172709_2 4,579,200 20,692 0.0045
09 May 2023 16:19:56 1478928 22297929 oifs_43r3_ps_0065_1987050100_123_956_12172709_2 3,110,400 13,630 0.0044
09 May 2023 16:19:56 1478928 22297929 oifs_43r3_ps_0065_1987050100_123_956_12172709_2 3,110,400 13,630 0.0044
09 May 2023 16:19:52 1478928 22297929 oifs_43r3_ps_0065_1987050100_123_956_12172709_2 1,728,000 7,077 0.0041
09 May 2023 16:19:52 1478928 22297929 oifs_43r3_ps_0065_1987050100_123_956_12172709_2 1,728,000 7,077 0.0041
09 May 2023 16:19:44 1478928 22297929 oifs_43r3_ps_0065_1987050100_123_956_12172709_2 864,000 3,243 0.0038
09 May 2023 16:19:44 1478928 22297929 oifs_43r3_ps_0065_1987050100_123_956_12172709_2 864,000 3,243 0.0038


©2024 climateprediction.net