climateprediction.net home page
Task 22287394

Task 22287394

Name oifs_43r3_ps_0475_2015050100_123_984_12201119_0
Workunit 12201119
Created 24 Dec 2022, 12:53:08 UTC
Sent 24 Dec 2022, 12:55:53 UTC
Report deadline 23 Jan 2023, 12:55:53 UTC
Received 28 Dec 2022, 18:08:56 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1524359
Run time 2 days 22 hours 30 min 25 sec
CPU time 11 hours 50 min 13 sec
Validate state Invalid
Credit 3,507.84
Device peak FLOPS 3.52 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,404.60 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,504.64 MB
Stderr 
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
00] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:5086:5086] [20221227:203953:1672202393:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:46:42 STEP  816 H= 816:00 +CPU= 39.621
  20:47:08 STEP  817 H= 817:00 +CPU= 20.383
  20:47:32 STEP  818 H= 818:00 +CPU= 24.099
  20:48:18 STEP  819 H= 819:00 +CPU= 43.651
  20:48:40 STEP  820 H= 820:00 +CPU= 21.471
  20:49:01 STEP  821 H= 821:00 +CPU= 20.755
  20:49:37 STEP  822 H= 822:00 +CPU= 36.362
  20:50:12 STEP  823 H= 823:00 +CPU= 34.698
  20:50:47 STEP  824 H= 824:00 +CPU= 35.600
  20:51:26 STEP  825 H= 825:00 +CPU= 38.245
  20:51:53 STEP  826 H= 826:00 +CPU= 27.047
  20:52:27 STEP  827 H= 827:00 +CPU= 34.526
  20:53:10 STEP  828 H= 828:00 +CPU= 41.606
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000828
  20:53:36 STEP  829 H= 829:00 +CPU= 26.265
  20:54:06 STEP  830 H= 830:00 +CPU= 29.896
  20:54:47 STEP  831 H= 831:00 +CPU= 41.384
  20:55:08 STEP  832 H= 832:00 +CPU= 21.150
  20:55:29 STEP  833 H= 833:00 +CPU= 20.543
  20:56:02 STEP  834 H= 834:00 +CPU= 33.248
  20:56:23 STEP  835 H= 835:00 +CPU= 20.549
  20:56:44 STEP  836 H= 836:00 +CPU= 21.096
  20:57:17 STEP  837 H= 837:00 +CPU= 32.614
  20:57:43 STEP  838 H= 838:00 +CPU= 26.116
  20:58:41 STEP  839 H= 839:00 +CPU= 27.772
  21:00:12 STEP  840 H= 840:00 +CPU= 62.778
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_34.zip
Uploading the intermediate file: upload_file_34.zip
  21:00:47 STEP  841 H= 841:00 +CPU= 32.821
  21:01:23 STEP  842 H= 842:00 +CPU= 36.113
  21:04:28 STEP  843 H= 843:00 +CPU= 55.305
  21:05:10 STEP  844 H= 844:00 +CPU= 36.737
  21:05:43 STEP  845 H= 845:00 +CPU= 32.551
  21:06:36 STEP  846 H= 846:00 +CPU= 51.507
  21:07:12 STEP  847 H= 847:00 +CPU= 35.674
  21:07:49 STEP  848 H= 848:00 +CPU= 36.824
  21:19:22 STEP  849 H= 849:00 +CPU= 54.253
  21:19:55 STEP  850 H= 850:00 +CPU= 32.127
  21:20:28 STEP  851 H= 851:00 +CPU= 31.445
  21:21:32 STEP  852 H= 852:00 +CPU= 61.615
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000852
  21:22:08 STEP  853 H= 853:00 +CPU= 35.822
  21:22:47 STEP  854 H= 854:00 +CPU= 36.522
  21:23:42 STEP  855 H= 855:00 +CPU= 53.548
  21:24:16 STEP  856 H= 856:00 +CPU= 33.459
  21:24:52 STEP  857 H= 857:00 +CPU= 35.802
  21:25:49 STEP  858 H= 858:00 +CPU= 56.764
  21:26:26 STEP  859 H= 859:00 +CPU= 35.935
  21:27:04 STEP  860 H= 860:00 +CPU= 36.774
  21:28:00 STEP  861 H= 861:00 +CPU= 55.622
  21:28:37 STEP  862 H= 862:00 +CPU= 36.890
  21:29:15 STEP  863 H= 863:00 +CPU= 36.645
  21:31:35 STEP  864 H= 864:00 +CPU= 61.407
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_35.zip
Uploading the intermediate file: upload_file_35.zip
  21:32:05 STEP  865 H= 865:00 +CPU= 29.431
  21:32:27 STEP  866 H= 866:00 +CPU= 21.382
  21:33:06 STEP  867 H= 867:00 +CPU= 32.203
  21:33:27 STEP  868 H= 868:00 +CPU= 20.940
  21:33:48 STEP  869 H= 869:00 +CPU= 20.369
  21:34:21 STEP  870 H= 870:00 +CPU= 32.921
  21:34:43 STEP  871 H= 871:00 +CPU= 21.400
  21:57:56 STEP  872 H= 872:00 +CPU= 40.819
  22:01:48 STEP  873 H= 873:00 +CPU= 37.917
  22:04:17 STEP  874 H= 874:00 +CPU= 24.072
  22:04:39 STEP  875 H= 875:00 +CPU= 20.182
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0475
(argv4) batchid: 984
(argv5) wuid: 12201119
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a15d5912f1051cdf19c62f8686fb4128 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0475_2015050100_123_984_12201119.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0475_2015050100_123_984_12201119.zip
ic_ancil_file: ic_ancil_12201119
ifsdata_file: ifsdata_12201119
climate_data_file: clim_data_12201119
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_22315c3977e7f92d0893fc294a21dfe8 to: /var/lib/boinc-client/slots/4/ic_ancil_12201119.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12201119.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fc911c0c734ed51c499d1c4ae19b0683 to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12201119.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12201119.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12201119.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12201119.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12201119.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12201119.xml
-- Model is restarting --
Adjusting last_iter, 876, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12201119.xml
last_cpu_time: 31312
upload_file_number: 36
last_iter: 863
last_upload: 3110400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147
The child process has been launched with process id: 10785
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:10785:10785] [20221227:222958:1672208998:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:31:40 STEP  864 H= 864:00 +CPU= 63.681
  22:33:09 STEP  865 H= 865:00 +CPU= 35.051
  22:33:45 STEP  866 H= 866:00 +CPU= 35.603
  22:34:40 STEP  867 H= 867:00 +CPU= 53.020
  22:35:13 STEP  868 H= 868:00 +CPU= 33.204
  22:35:40 STEP  869 H= 869:00 +CPU= 26.916
  22:36:25 STEP  870 H= 870:00 +CPU= 45.162
  22:36:46 STEP  871 H= 871:00 +CPU= 20.652
  22:37:07 STEP  872 H= 872:00 +CPU= 21.272
  22:37:58 STEP  873 H= 873:00 +CPU= 51.359
  22:38:34 STEP  874 H= 874:00 +CPU= 35.857
  22:39:09 STEP  875 H= 875:00 +CPU= 34.806
  22:40:44 STEP  876 H= 876:00 +CPU= 55.640
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000876
  22:41:24 STEP  877 H= 877:00 +CPU= 35.607
  22:42:01 STEP  878 H= 878:00 +CPU= 34.455
  22:43:01 STEP  879 H= 879:00 +CPU= 56.174
  22:43:39 STEP  880 H= 880:00 +CPU= 37.359
  22:44:15 STEP  881 H= 881:00 +CPU= 35.401
  22:45:09 STEP  882 H= 882:00 +CPU= 51.526
  22:45:42 STEP  883 H= 883:00 +CPU= 31.972
  22:46:17 STEP  884 H= 884:00 +CPU= 34.760
  22:47:13 STEP  885 H= 885:00 +CPU= 55.894
  22:47:51 STEP  886 H= 886:00 +CPU= 37.410
  22:48:28 STEP  887 H= 887:00 +CPU= 35.876
  22:49:43 STEP  888 H= 888:00 +CPU= 62.477
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_36.zip
Uploading the intermediate file: upload_file_36.zip
  22:50:17 STEP  889 H= 889:00 +CPU= 33.339
  22:50:51 STEP  890 H= 890:00 +CPU= 33.301
  22:51:42 STEP  891 H= 891:00 +CPU= 49.935
  22:52:16 STEP  892 H= 892:00 +CPU= 34.463
  22:52:53 STEP  893 H= 893:00 +CPU= 36.075
  22:53:44 STEP  894 H= 894:00 +CPU= 50.424
  22:57:55 STEP  895 H= 895:00 +CPU= 36.315
  22:58:32 STEP  896 H= 896:00 +CPU= 36.074
  22:59:27 STEP  897 H= 897:00 +CPU= 54.421
  23:00:04 STEP  898 H= 898:00 +CPU= 36.020
  23:00:39 STEP  899 H= 899:00 +CPU= 34.730
  23:01:41 STEP  900 H= 900:00 +CPU= 60.518
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000900
  23:02:17 STEP  901 H= 901:00 +CPU= 34.819
  23:02:53 STEP  902 H= 902:00 +CPU= 36.405
  23:03:43 STEP  903 H= 903:00 +CPU= 48.606
  23:08:17 STEP  904 H= 904:00 +CPU= 38.141
  23:09:52 STEP  905 H= 905:00 +CPU= 34.502
  23:10:49 STEP  906 H= 906:00 +CPU= 53.874
  23:11:26 STEP  907 H= 907:00 +CPU= 36.113
  23:12:05 STEP  908 H= 908:00 +CPU= 37.282
  23:13:01 STEP  909 H= 909:00 +CPU= 55.889
  23:13:35 STEP  910 H= 910:00 +CPU= 33.128
  23:14:08 STEP  911 H= 911:00 +CPU= 32.344
  23:16:39 STEP  912 H= 912:00 +CPU= 65.303
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_37.zip
Uploading the intermediate file: upload_file_37.zip
  23:17:17 STEP  913 H= 913:00 +CPU= 34.946
  23:17:55 STEP  914 H= 914:00 +CPU= 37.179
  23:18:47 STEP  915 H= 915:00 +CPU= 51.372
  23:19:23 STEP  916 H= 916:00 +CPU= 35.099
  23:19:59 STEP  917 H= 917:00 +CPU= 35.862
  23:20:57 STEP  918 H= 918:00 +CPU= 57.382
  23:21:33 STEP  919 H= 919:00 +CPU= 35.842
  23:22:11 STEP  920 H= 920:00 +CPU= 37.225
  23:23:08 STEP  921 H= 921:00 +CPU= 56.139
  23:23:43 STEP  922 H= 922:00 +CPU= 34.795
  23:24:14 STEP  923 H= 923:00 +CPU= 29.658
  23:25:45 STEP  924 H= 924:00 +CPU= 61.961
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000924
  23:26:24 STEP  925 H= 925:00 +CPU= 35.862
  23:27:02 STEP  926 H= 926:00 +CPU= 37.248
  23:28:02 STEP  927 H= 927:00 +CPU= 56.350
  23:28:40 STEP  928 H= 928:00 +CPU= 37.227
  23:29:17 STEP  929 H= 929:00 +CPU= 35.797
  23:30:15 STEP  930 H= 930:00 +CPU= 53.783
  23:30:47 STEP  931 H= 931:00 +CPU= 31.957
  23:31:21 STEP  932 H= 932:00 +CPU= 33.169
  23:32:12 STEP  933 H= 933:00 +CPU= 50.138
  23:32:45 STEP  934 H= 934:00 +CPU= 33.146
  23:33:20 STEP  935 H= 935:00 +CPU= 33.475
  23:36:09 STEP  936 H= 936:00 +CPU= 63.886
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_38.zip
Uploading the intermediate file: upload_file_38.zip
  23:36:45 STEP  937 H= 937:00 +CPU= 34.298
  23:38:01 STEP  938 H= 938:00 +CPU= 36.943
  23:39:34 STEP  939 H= 939:00 +CPU= 55.316
  23:40:09 STEP  940 H= 940:00 +CPU= 34.578
  23:40:45 STEP  941 H= 941:00 +CPU= 35.868
  23:41:43 STEP  942 H= 942:00 +CPU= 57.015
  23:42:18 STEP  943 H= 943:00 +CPU= 34.075
  23:42:55 STEP  944 H= 944:00 +CPU= 37.060
  23:43:51 STEP  945 H= 945:00 +CPU= 56.159
  23:44:29 STEP  946 H= 946:00 +CPU= 36.912
  23:45:04 STEP  947 H= 947:00 +CPU= 35.560
  23:46:04 STEP  948 H= 948:00 +CPU= 57.136
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000948
  23:46:34 STEP  949 H= 949:00 +CPU= 29.593
  23:47:12 STEP  950 H= 950:00 +CPU= 37.023
  23:48:08 STEP  951 H= 951:00 +CPU= 55.474
  23:48:46 STEP  952 H= 952:00 +CPU= 37.083
  23:49:58 STEP  953 H= 953:00 +CPU= 29.071
  23:50:55 STEP  954 H= 954:00 +CPU= 56.232
  23:51:30 STEP  955 H= 955:00 +CPU= 33.501
  23:52:04 STEP  956 H= 956:00 +CPU= 32.992
  23:52:54 STEP  957 H= 957:00 +CPU= 49.920
  23:53:28 STEP  958 H= 958:00 +CPU= 34.023
  23:54:07 STEP  959 H= 959:00 +CPU= 37.274
  23:55:22 STEP  960 H= 960:00 +CPU= 58.559
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_39.zip
Uploading the intermediate file: upload_file_39.zip
  23:55:58 STEP  961 H= 961:00 +CPU= 34.362
  23:56:35 STEP  962 H= 962:00 +CPU= 36.658
  00:27:06 STEP  963 H= 963:00 +CPU= 52.051
  00:30:41 STEP  964 H= 964:00 +CPU= 37.476
  00:32:16 STEP  965 H= 965:00 +CPU= 35.183
  00:33:17 STEP  966 H= 966:00 +CPU= 56.991
  00:33:53 STEP  967 H= 967:00 +CPU= 35.363
  00:34:32 STEP  968 H= 968:00 +CPU= 36.783
  00:35:25 STEP  969 H= 969:00 +CPU= 51.608
  00:36:02 STEP  970 H= 970:00 +CPU= 36.791
  00:37:12 STEP  971 H= 971:00 +CPU= 33.250
  00:38:18 STEP  972 H= 972:00 +CPU= 60.979
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000972
  00:38:54 STEP  973 H= 973:00 +CPU= 35.713
  00:39:32 STEP  974 H= 974:00 +CPU= 36.755
  00:40:30 STEP  975 H= 975:00 +CPU= 55.468
  00:41:07 STEP  976 H= 976:00 +CPU= 36.895
  00:41:43 STEP  977 H= 977:00 +CPU= 35.548
  00:42:41 STEP  978 H= 978:00 +CPU= 56.390
  00:43:17 STEP  979 H= 979:00 +CPU= 35.371
  00:43:52 STEP  980 H= 980:00 +CPU= 34.008
  00:44:46 STEP  981 H= 981:00 +CPU= 52.665
  00:45:23 STEP  982 H= 982:00 +CPU= 36.537
  00:46:01 STEP  983 H= 983:00 +CPU= 36.843
  00:47:35 STEP  984 H= 984:00 +CPU= 51.358
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_40.zip
Uploading the intermediate file: upload_file_40.zip
  01:26:10 STEP  985 H= 985:00 +CPU= 36.846
  02:42:53 STEP  986 H= 986:00 +CPU= 46.649
  04:15:21 STEP  987 H= 987:00 +CPU= 67.116
  04:22:38 STEP  988 H= 988:00 +CPU= 40.379
  04:23:17 STEP  989 H= 989:00 +CPU= 36.680
  04:24:19 STEP  990 H= 990:00 +CPU= 55.573
  04:25:40 STEP  991 H= 991:00 +CPU= 35.545
  04:26:19 STEP  992 H= 992:00 +CPU= 37.194
  04:27:17 STEP  993 H= 993:00 +CPU= 56.498
  04:27:56 STEP  994 H= 994:00 +CPU= 37.550
  04:28:31 STEP  995 H= 995:00 +CPU= 34.316
  04:29:35 STEP  996 H= 996:00 +CPU= 59.676
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000996
  04:32:02 STEP  997 H= 997:00 +CPU= 35.101
  04:32:41 STEP  998 H= 998:00 +CPU= 37.687
  04:33:38 STEP  999 H= 999:00 +CPU= 56.036
  04:34:12 STEP 1000 H=1000:00 +CPU= 33.264
  04:34:45 STEP 1001 H=1001:00 +CPU= 32.068
  04:35:37 STEP 1002 H=1002:00 +CPU= 51.647
  04:36:10 STEP 1003 H=1003:00 +CPU= 32.243
  04:36:44 STEP 1004 H=1004:00 +CPU= 33.052
  04:37:35 STEP 1005 H=1005:00 +CPU= 40.833
  04:48:27 STEP 1006 H=1006:00 +CPU= 29.145
  04:51:17 STEP 1007 H=1007:00 +CPU= 39.243
  04:54:03 STEP 1008 H=1008:00 +CPU= 64.352
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_41.zip
Uploading the intermediate file: upload_file_41.zip
  04:54:31 STEP 1009 H=1009:00 +CPU= 26.686
  04:55:12 STEP 1010 H=1010:00 +CPU= 36.288
  04:56:02 STEP 1011 H=1011:00 +CPU= 49.119
  04:58:15 STEP 1012 H=1012:00 +CPU= 35.861
  04:58:52 STEP 1013 H=1013:00 +CPU= 34.259
  05:01:12 STEP 1014 H=1014:00 +CPU= 55.680
  05:01:48 STEP 1015 H=1015:00 +CPU= 34.377
  05:02:26 STEP 1016 H=1016:00 +CPU= 37.171
  05:03:21 STEP 1017 H=1017:00 +CPU= 52.827
  05:03:54 STEP 1018 H=1018:00 +CPU= 32.526
  05:04:30 STEP 1019 H=1019:00 +CPU= 34.716
  05:06:23 STEP 1020 H=1020:00 +CPU= 62.587
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001020
  05:07:00 STEP 1021 H=1021:00 +CPU= 35.575
  05:07:37 STEP 1022 H=1022:00 +CPU= 36.640
  05:08:34 STEP 1023 H=1023:00 +CPU= 55.691
  05:09:10 STEP 1024 H=1024:00 +CPU= 34.695
  05:09:43 STEP 1025 H=1025:00 +CPU= 33.309
  05:10:36 STEP 1026 H=1026:00 +CPU= 51.526
  05:11:11 STEP 1027 H=1027:00 +CPU= 34.439
  05:11:49 STEP 1028 H=1028:00 +CPU= 37.004
  05:12:46 STEP 1029 H=1029:00 +CPU= 55.754
  05:13:23 STEP 1030 H=1030:00 +CPU= 36.787
  05:14:00 STEP 1031 H=1031:00 +CPU= 36.670
  05:15:13 STEP 1032 H=1032:00 +CPU= 58.963
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_42.zip
Uploading the intermediate file: upload_file_42.zip
  05:15:47 STEP 1033 H=1033:00 +CPU= 29.900
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0475
(argv4) batchid: 984
(argv5) wuid: 12201119
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a15d5912f1051cdf19c62f8686fb4128 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0475_2015050100_123_984_12201119.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0475_2015050100_123_984_12201119.zip
ic_ancil_file: ic_ancil_12201119
ifsdata_file: ifsdata_12201119
climate_data_file: clim_data_12201119
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_22315c3977e7f92d0893fc294a21dfe8 to: /var/lib/boinc-client/slots/4/ic_ancil_12201119.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12201119.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fc911c0c734ed51c499d1c4ae19b0683 to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12201119.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12201119.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12201119.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12201119.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12201119.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12201119.xml
-- Model is restarting --
Adjusting last_iter, 1034, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12201119.xml
last_cpu_time: 38427
upload_file_number: 43
last_iter: 1031
last_upload: 3715200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147
The child process has been launched with process id: 32590
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:32590:32590] [20221228:053935:1672234775:0.053] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:47:25 STEP 1032 H=1032:00 +CPU= 39.899
  05:47:47 STEP 1033 H=1033:00 +CPU= 20.268
  05:48:07 STEP 1034 H=1034:00 +CPU= 20.475
  05:48:40 STEP 1035 H=1035:00 +CPU= 31.737
  05:49:01 STEP 1036 H=1036:00 +CPU= 20.486
  05:49:21 STEP 1037 H=1037:00 +CPU= 19.961
  05:49:53 STEP 1038 H=1038:00 +CPU= 32.440
  05:50:13 STEP 1039 H=1039:00 +CPU= 19.986
  05:50:34 STEP 1040 H=1040:00 +CPU= 20.520
  05:51:05 STEP 1041 H=1041:00 +CPU= 31.806
  05:51:26 STEP 1042 H=1042:00 +CPU= 20.555
  05:51:46 STEP 1043 H=1043:00 +CPU= 19.931
  05:52:28 STEP 1044 H=1044:00 +CPU= 35.149
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001044
  05:52:48 STEP 1045 H=1045:00 +CPU= 19.970
  05:53:09 STEP 1046 H=1046:00 +CPU= 20.556
  05:53:44 STEP 1047 H=1047:00 +CPU= 31.691
  05:54:06 STEP 1048 H=1048:00 +CPU= 20.615
  05:54:27 STEP 1049 H=1049:00 +CPU= 19.996
  05:55:02 STEP 1050 H=1050:00 +CPU= 32.437
  05:55:23 STEP 1051 H=1051:00 +CPU= 20.273
  05:55:44 STEP 1052 H=1052:00 +CPU= 21.008
  05:56:52 STEP 1053 H=1053:00 +CPU= 32.407
  05:57:26 STEP 1054 H=1054:00 +CPU= 20.668
  05:57:57 STEP 1055 H=1055:00 +CPU= 20.747
  06:00:01 STEP 1056 H=1056:00 +CPU= 37.712
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_43.zip
Uploading the intermediate file: upload_file_43.zip
  06:00:21 STEP 1057 H=1057:00 +CPU= 20.110
  06:00:42 STEP 1058 H=1058:00 +CPU= 20.946
  06:01:20 STEP 1059 H=1059:00 +CPU= 37.454
  06:01:50 STEP 1060 H=1060:00 +CPU= 29.966
  06:02:24 STEP 1061 H=1061:00 +CPU= 33.528
  06:03:09 STEP 1062 H=1062:00 +CPU= 45.615
  06:03:43 STEP 1063 H=1063:00 +CPU= 34.257
  06:04:17 STEP 1064 H=1064:00 +CPU= 33.089
  06:05:11 STEP 1065 H=1065:00 +CPU= 54.076
  06:05:43 STEP 1066 H=1066:00 +CPU= 31.904
  06:06:13 STEP 1067 H=1067:00 +CPU= 30.392
  06:07:11 STEP 1068 H=1068:00 +CPU= 58.096
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001068
  06:07:32 STEP 1069 H=1069:00 +CPU= 20.708
  06:07:53 STEP 1070 H=1070:00 +CPU= 21.169
  06:08:38 STEP 1071 H=1071:00 +CPU= 45.292
  06:09:46 STEP 1072 H=1072:00 +CPU= 34.468
  06:10:19 STEP 1073 H=1073:00 +CPU= 32.048
  06:11:51 STEP 1074 H=1074:00 +CPU= 55.714
  06:12:39 STEP 1075 H=1075:00 +CPU= 31.214
  06:13:10 STEP 1076 H=1076:00 +CPU= 30.956
  06:16:17 STEP 1077 H=1077:00 +CPU= 55.953
  06:16:47 STEP 1078 H=1078:00 +CPU= 24.078
  06:18:18 STEP 1079 H=1079:00 +CPU= 34.489
  06:20:04 STEP 1080 H=1080:00 +CPU= 65.341
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_44.zip
Uploading the intermediate file: upload_file_44.zip
  06:20:34 STEP 1081 H=1081:00 +CPU= 28.021
  06:24:01 STEP 1082 H=1082:00 +CPU= 23.887
  06:27:05 STEP 1083 H=1083:00 +CPU= 44.288
  06:30:30 STEP 1084 H=1084:00 +CPU= 37.948
  06:31:42 STEP 1085 H=1085:00 +CPU= 33.946
  06:34:54 STEP 1086 H=1086:00 +CPU= 55.950
  06:35:28 STEP 1087 H=1087:00 +CPU= 33.186
  06:36:06 STEP 1088 H=1088:00 +CPU= 36.944
  06:37:02 STEP 1089 H=1089:00 +CPU= 55.543
  06:37:40 STEP 1090 H=1090:00 +CPU= 36.983
  06:38:16 STEP 1091 H=1091:00 +CPU= 35.691
  06:41:50 STEP 1092 H=1092:00 +CPU= 57.422
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001092
  06:42:26 STEP 1093 H=1093:00 +CPU= 35.575
  06:43:03 STEP 1094 H=1094:00 +CPU= 36.785
  06:44:43 STEP 1095 H=1095:00 +CPU= 52.822
  06:45:20 STEP 1096 H=1096:00 +CPU= 36.559
  06:45:56 STEP 1097 H=1097:00 +CPU= 35.530
  06:46:54 STEP 1098 H=1098:00 +CPU= 56.892
  06:47:30 STEP 1099 H=1099:00 +CPU= 35.492
  06:48:09 STEP 1100 H=1100:00 +CPU= 36.819
  06:49:06 STEP 1101 H=1101:00 +CPU= 55.714
  06:49:43 STEP 1102 H=1102:00 +CPU= 36.603
  06:50:22 STEP 1103 H=1103:00 +CPU= 36.242
  06:51:44 STEP 1104 H=1104:00 +CPU= 64.324
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147_45.zip
Uploading the intermediate file: upload_file_45.zip
  06:52:17 STEP 1105 H=1105:00 +CPU= 32.036
  06:52:54 STEP 1106 H=1106:00 +CPU= 36.324
  06:53:47 STEP 1107 H=1107:00 +CPU= 52.100
  06:54:24 STEP 1108 H=1108:00 +CPU= 36.719
  06:55:00 STEP 1109 H=1109:00 +CPU= 35.667
  06:55:51 STEP 1110 H=1110:00 +CPU= 50.495
  06:56:27 STEP 1111 H=1111:00 +CPU= 35.005
  06:57:00 STEP 1112 H=1112:00 +CPU= 32.495
  06:57:56 STEP 1113 H=1113:00 +CPU= 54.932
  06:58:33 STEP 1114 H=1114:00 +CPU= 36.817
  06:59:10 STEP 1115 H=1115:00 +CPU= 35.850
  07:00:14 STEP 1116 H=1116:00 +CPU= 62.315
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001116
  07:00:50 STEP 1117 H=1117:00 +CPU= 35.722
  07:01:27 STEP 1118 H=1118:00 +CPU= 35.743
  07:30:49 STEP 1119 H=1119:00 +CPU= 55.959
  08:26:38 STEP 1120 H=1120:00 +CPU= 46.578
  08:28:44 STEP 1121 H=1121:00 +CPU= 34.531
  08:31:00 STEP 1122 H=1122:00 +CPU= 37.387
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0475
(argv4) batchid: 984
(argv5) wuid: 12201119
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201119
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a15d5912f1051cdf19c62f8686fb4128 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0475_2015050100_123_984_12201119.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0475_2015050100_123_984_12201119.zip
ic_ancil_file: ic_ancil_12201119
ifsdata_file: ifsdata_12201119
climate_data_file: clim_data_12201119
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_22315c3977e7f92d0893fc294a21dfe8 to: /var/lib/boinc-client/slots/4/ic_ancil_12201119.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12201119.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fc911c0c734ed51c499d1c4ae19b0683 to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12201119.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12201119.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12201119.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12201119.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12201119.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12201119.xml
-- Model is restarting --
Adjusting last_iter, 1123, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12201119.xml
last_cpu_time: 41694
upload_file_number: 46
last_iter: 1103
last_upload: 3974400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0475_2015050100_123_984_12201119_0_r1431041147
The child process has been launched with process id: 9425
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:9425:9425] [20221228:084737:1672246057:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:49:23 STEP 1104 H=1104:00 +CPU= 57.460
  08:52:36 STEP 1105 H=1105:00 +CPU= 29.743
  08:53:13 STEP 1106 H=1106:00 +CPU= 35.965
  08:54:07 STEP 1107 H=1107:00 +CPU= 52.945
  08:54:32 STEP 1108 H=1108:00 +CPU= 24.866
  08:54:53 STEP 1109 H=1109:00 +CPU= 20.319
  08:55:35 STEP 1110 H=1110:00 +CPU= 42.394
  08:55:56 STEP 1111 H=1111:00 +CPU= 20.999
  08:56:17 STEP 1112 H=1112:00 +CPU= 20.912
  08:56:55 STEP 1113 H=1113:00 +CPU= 37.512
  08:57:17 STEP 1114 H=1114:00 +CPU= 22.484
  08:57:51 STEP 1115 H=1115:00 +CPU= 34.151
  08:58:53 STEP 1116 H=1116:00 +CPU= 60.445
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001116
  09:00:19 STEP 1117 H=1117:00 +CPU= 33.176
  09:00:55 STEP 1118 H=1118:00 +CPU= 35.494
  09:01:50 STEP 1119 H=1119:00 +CPU= 49.372
  09:02:27 STEP 1120 H=1120:00 +CPU= 33.902
  09:03:02 STEP 1121 H=1121:00 +CPU= 35.398
  09:04:01 STEP 1122 H=1122:00 +CPU= 56.727
  09:04:37 STEP 1123 H=1123:00 +CPU= 35.444
  09:05:14 STEP 1124 H=1124:00 +CPU= 36.308
  09:09:22 STEP 1125 H=1125:00 +CPU= 50.619
  09:11:14 STEP 1126 H=1126:00 +CPU= 36.426
  09:12:14 STEP 1127 H=1127:00 +CPU= 30.967
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:58:21 1524359 22287394 oifs_43r3_ps_0475_2015050100_123_984_12201119_0 2,505,600 23,883 0.0095
09 May 2023 15:57:07 1524359 22287394 oifs_43r3_ps_0475_2015050100_123_984_12201119_0 1,641,600 17,011 0.0104


©2024 cpdn.org