<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
SHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351_35.zip
Uploading the intermediate file: upload_file_35.zip
11:11:58 STEP 865 H= 865:00 +CPU= 20.331
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:13:20 STEP 866 H= 866:00 +CPU= 21.129
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:14:44 STEP 867 H= 867:00 +CPU= 32.993
11:15:06 STEP 868 H= 868:00 +CPU= 22.059
11:15:26 STEP 869 H= 869:00 +CPU= 19.951
11:16:00 STEP 870 H= 870:00 +CPU= 33.394
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:17:13 STEP 871 H= 871:00 +CPU= 20.827
11:17:33 STEP 872 H= 872:00 +CPU= 20.264
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:18:59 STEP 873 H= 873:00 +CPU= 33.355
11:19:20 STEP 874 H= 874:00 +CPU= 21.210
11:19:40 STEP 875 H= 875:00 +CPU= 20.056
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:21:08 STEP 876 H= 876:00 +CPU= 36.452
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000876
11:21:29 STEP 877 H= 877:00 +CPU= 20.794
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:22:32 STEP 878 H= 878:00 +CPU= 21.352
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:23:45 STEP 879 H= 879:00 +CPU= 32.897
11:24:06 STEP 880 H= 880:00 +CPU= 20.607
11:24:26 STEP 881 H= 881:00 +CPU= 19.745
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:25:50 STEP 882 H= 882:00 +CPU= 33.399
11:26:10 STEP 883 H= 883:00 +CPU= 20.241
11:26:31 STEP 884 H= 884:00 +CPU= 20.487
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:27:54 STEP 885 H= 885:00 +CPU= 32.076
11:28:15 STEP 886 H= 886:00 +CPU= 20.641
11:28:36 STEP 887 H= 887:00 +CPU= 20.612
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:30:06 STEP 888 H= 888:00 +CPU= 38.713
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351_36.zip
Uploading the intermediate file: upload_file_36.zip
11:30:27 STEP 889 H= 889:00 +CPU= 20.883
11:30:47 STEP 890 H= 890:00 +CPU= 20.479
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:32:32 STEP 891 H= 891:00 +CPU= 33.089
11:32:53 STEP 892 H= 892:00 +CPU= 20.570
11:33:13 STEP 893 H= 893:00 +CPU= 20.187
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0626
(argv4) batchid: 972
(argv5) wuid: 12189270
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d8f54b1e4a7f06db78a22b9834c4bfff to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0626_2003050100_123_972_12189270.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0626_2003050100_123_972_12189270.zip
ic_ancil_file: ic_ancil_12189270
ifsdata_file: ifsdata_12189270
climate_data_file: clim_data_12189270
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_87ccf1a7e3970821feeeddf963e21501 to: /var/lib/boinc-client/slots/3/ic_ancil_12189270.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12189270.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_54f804466577be82f0c91bae5dd17ae8 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12189270.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12189270.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12189270.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12189270.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12189270.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12189270.xml
-- Model is restarting --
Adjusting last_iter, 894, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12189270.xml
last_cpu_time: 23848
upload_file_number: 37
last_iter: 887
last_upload: 3196800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351
The child process has been launched with process id: 922358
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:922358:922358] [20230104:113437:1672799677:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:35:58 STEP 888 H= 888:00 +CPU= 36.433
11:36:19 STEP 889 H= 889:00 +CPU= 21.243
11:36:40 STEP 890 H= 890:00 +CPU= 21.268
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:38:25 STEP 891 H= 891:00 +CPU= 33.006
11:38:46 STEP 892 H= 892:00 +CPU= 20.965
11:39:06 STEP 893 H= 893:00 +CPU= 20.354
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:40:31 STEP 894 H= 894:00 +CPU= 33.419
11:40:51 STEP 895 H= 895:00 +CPU= 20.165
11:41:12 STEP 896 H= 896:00 +CPU= 21.129
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:42:46 STEP 897 H= 897:00 +CPU= 33.023
11:43:07 STEP 898 H= 898:00 +CPU= 20.807
11:43:28 STEP 899 H= 899:00 +CPU= 20.388
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:45:37 STEP 900 H= 900:00 +CPU= 36.451
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000900
11:45:58 STEP 901 H= 901:00 +CPU= 20.862
11:46:19 STEP 902 H= 902:00 +CPU= 20.984
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:47:44 STEP 903 H= 903:00 +CPU= 33.298
11:48:05 STEP 904 H= 904:00 +CPU= 20.921
11:48:25 STEP 905 H= 905:00 +CPU= 20.085
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:49:50 STEP 906 H= 906:00 +CPU= 33.712
11:50:10 STEP 907 H= 907:00 +CPU= 19.862
11:50:31 STEP 908 H= 908:00 +CPU= 20.982
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:51:45 STEP 909 H= 909:00 +CPU= 32.804
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:52:57 STEP 910 H= 910:00 +CPU= 20.595
11:53:19 STEP 911 H= 911:00 +CPU= 21.916
11:53:57 STEP 912 H= 912:00 +CPU= 37.737
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351_37.zip
Uploading the intermediate file: upload_file_37.zip
11:54:17 STEP 913 H= 913:00 +CPU= 19.895
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:55:31 STEP 914 H= 914:00 +CPU= 20.492
11:56:04 STEP 915 H= 915:00 +CPU= 32.558
11:56:25 STEP 916 H= 916:00 +CPU= 20.691
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:57:46 STEP 917 H= 917:00 +CPU= 20.911
11:58:19 STEP 918 H= 918:00 +CPU= 33.500
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:59:30 STEP 919 H= 919:00 +CPU= 20.094
11:59:51 STEP 920 H= 920:00 +CPU= 21.521
12:00:24 STEP 921 H= 921:00 +CPU= 32.610
12:00:45 STEP 922 H= 922:00 +CPU= 20.906
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:02:06 STEP 923 H= 923:00 +CPU= 20.652
12:02:42 STEP 924 H= 924:00 +CPU= 36.405
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000924
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:03:55 STEP 925 H= 925:00 +CPU= 20.277
12:04:16 STEP 926 H= 926:00 +CPU= 20.936
12:04:48 STEP 927 H= 927:00 +CPU= 32.388
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:06:11 STEP 928 H= 928:00 +CPU= 21.071
12:06:31 STEP 929 H= 929:00 +CPU= 20.228
12:07:04 STEP 930 H= 930:00 +CPU= 33.091
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:08:16 STEP 931 H= 931:00 +CPU= 20.793
12:08:37 STEP 932 H= 932:00 +CPU= 20.747
12:09:09 STEP 933 H= 933:00 +CPU= 31.764
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:10:21 STEP 934 H= 934:00 +CPU= 21.004
12:10:42 STEP 935 H= 935:00 +CPU= 20.765
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:12:31 STEP 936 H= 936:00 +CPU= 38.184
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351_38.zip
Uploading the intermediate file: upload_file_38.zip
12:12:52 STEP 937 H= 937:00 +CPU= 21.226
12:13:13 STEP 938 H= 938:00 +CPU= 20.508
12:13:46 STEP 939 H= 939:00 +CPU= 32.518
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:14:49 STEP 940 H= 940:00 +CPU= 21.264
12:15:10 STEP 941 H= 941:00 +CPU= 20.504
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:16:44 STEP 942 H= 942:00 +CPU= 33.405
12:17:04 STEP 943 H= 943:00 +CPU= 20.269
12:17:26 STEP 944 H= 944:00 +CPU= 21.264
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:18:42 STEP 945 H= 945:00 +CPU= 33.690
12:19:03 STEP 946 H= 946:00 +CPU= 21.176
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:20:13 STEP 947 H= 947:00 +CPU= 20.293
12:20:50 STEP 948 H= 948:00 +CPU= 36.099
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000948
12:21:10 STEP 949 H= 949:00 +CPU= 19.758
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:22:33 STEP 950 H= 950:00 +CPU= 20.984
12:23:06 STEP 951 H= 951:00 +CPU= 32.720
12:23:26 STEP 952 H= 952:00 +CPU= 20.333
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:24:39 STEP 953 H= 953:00 +CPU= 20.939
12:25:12 STEP 954 H= 954:00 +CPU= 33.376
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:26:25 STEP 955 H= 955:00 +CPU= 20.151
12:26:46 STEP 956 H= 956:00 +CPU= 21.404
12:27:18 STEP 957 H= 957:00 +CPU= 32.121
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:28:41 STEP 958 H= 958:00 +CPU= 20.835
12:29:03 STEP 959 H= 959:00 +CPU= 21.033
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:30:31 STEP 960 H= 960:00 +CPU= 37.881
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351_39.zip
Uploading the intermediate file: upload_file_39.zip
12:30:52 STEP 961 H= 961:00 +CPU= 20.830
12:31:13 STEP 962 H= 962:00 +CPU= 21.183
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:32:48 STEP 963 H= 963:00 +CPU= 33.341
12:33:08 STEP 964 H= 964:00 +CPU= 20.324
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:34:11 STEP 965 H= 965:00 +CPU= 21.413
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:35:35 STEP 966 H= 966:00 +CPU= 33.395
12:35:55 STEP 967 H= 967:00 +CPU= 19.596
12:36:16 STEP 968 H= 968:00 +CPU= 20.985
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:38:00 STEP 969 H= 969:00 +CPU= 32.931
12:38:21 STEP 970 H= 970:00 +CPU= 20.522
12:38:41 STEP 971 H= 971:00 +CPU= 20.264
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:40:19 STEP 972 H= 972:00 +CPU= 35.861
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000972
12:40:39 STEP 973 H= 973:00 +CPU= 19.760
12:41:00 STEP 974 H= 974:00 +CPU= 20.758
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:42:43 STEP 975 H= 975:00 +CPU= 32.524
12:43:04 STEP 976 H= 976:00 +CPU= 20.684
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:44:16 STEP 977 H= 977:00 +CPU= 20.053
12:44:50 STEP 978 H= 978:00 +CPU= 33.488
12:45:10 STEP 979 H= 979:00 +CPU= 19.918
12:45:31 STEP 980 H= 980:00 +CPU= 20.657
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:47:37 STEP 981 H= 981:00 +CPU= 34.806
12:48:01 STEP 982 H= 982:00 +CPU= 23.964
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:53:59 STEP 983 H= 983:00 +CPU= 24.885
12:54:45 STEP 984 H= 984:00 +CPU= 44.285
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351_40.zip
Uploading the intermediate file: upload_file_40.zip
12:55:08 STEP 985 H= 985:00 +CPU= 23.255
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:03:23 STEP 986 H= 986:00 +CPU= 24.387
13:04:00 STEP 987 H= 987:00 +CPU= 37.141
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:11:35 STEP 988 H= 988:00 +CPU= 22.998
13:11:57 STEP 989 H= 989:00 +CPU= 21.897
13:12:31 STEP 990 H= 990:00 +CPU= 33.748
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:15:14 STEP 991 H= 991:00 +CPU= 21.421
13:15:38 STEP 992 H= 992:00 +CPU= 23.231
13:16:15 STEP 993 H= 993:00 +CPU= 36.189
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:18:10 STEP 994 H= 994:00 +CPU= 23.252
13:18:33 STEP 995 H= 995:00 +CPU= 22.571
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:20:54 STEP 996 H= 996:00 +CPU= 39.767
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000996
13:21:17 STEP 997 H= 997:00 +CPU= 22.259
13:21:39 STEP 998 H= 998:00 +CPU= 21.578
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:23:37 STEP 999 H= 999:00 +CPU= 36.345
13:24:00 STEP 1000 H=1000:00 +CPU= 23.182
13:24:23 STEP 1001 H=1001:00 +CPU= 22.196
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:27:02 STEP 1002 H=1002:00 +CPU= 36.845
13:27:24 STEP 1003 H=1003:00 +CPU= 22.336
13:27:49 STEP 1004 H=1004:00 +CPU= 23.975
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:29:57 STEP 1005 H=1005:00 +CPU= 36.470
13:30:21 STEP 1006 H=1006:00 +CPU= 23.468
13:30:45 STEP 1007 H=1007:00 +CPU= 23.186
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:32:59 STEP 1008 H=1008:00 +CPU= 42.788
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351_41.zip
Uploading the intermediate file: upload_file_41.zip
13:33:20 STEP 1009 H=1009:00 +CPU= 20.986
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:53:35 STEP 1010 H=1010:00 +CPU= 23.152
13:54:15 STEP 1011 H=1011:00 +CPU= 38.496
13:54:41 STEP 1012 H=1012:00 +CPU= 25.162
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0626
(argv4) batchid: 972
(argv5) wuid: 12189270
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d8f54b1e4a7f06db78a22b9834c4bfff to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0626_2003050100_123_972_12189270.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0626_2003050100_123_972_12189270.zip
ic_ancil_file: ic_ancil_12189270
ifsdata_file: ifsdata_12189270
climate_data_file: clim_data_12189270
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_87ccf1a7e3970821feeeddf963e21501 to: /var/lib/boinc-client/slots/3/ic_ancil_12189270.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12189270.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_54f804466577be82f0c91bae5dd17ae8 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12189270.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12189270.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12189270.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12189270.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12189270.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12189270.xml
-- Model is restarting --
Adjusting last_iter, 1013, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12189270.xml
last_cpu_time: 27098
upload_file_number: 42
last_iter: 1007
last_upload: 3628800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351
The child process has been launched with process id: 948389
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:948389:948389] [20230104:141805:1672809485:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
14:18:54 STEP 1008 H=1008:00 +CPU= 42.852
14:19:19 STEP 1009 H=1009:00 +CPU= 23.906
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:29:38 STEP 1010 H=1010:00 +CPU= 25.907
14:30:17 STEP 1011 H=1011:00 +CPU= 38.389
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:41:25 STEP 1012 H=1012:00 +CPU= 24.985
14:41:51 STEP 1013 H=1013:00 +CPU= 24.639
14:42:30 STEP 1014 H=1014:00 +CPU= 38.682
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:09:29 STEP 1015 H=1015:00 +CPU= 24.303
15:09:56 STEP 1016 H=1016:00 +CPU= 26.115
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:26:19 STEP 1017 H=1017:00 +CPU= 39.170
15:26:40 STEP 1018 H=1018:00 +CPU= 21.660
15:27:00 STEP 1019 H=1019:00 +CPU= 19.914
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:28:48 STEP 1020 H=1020:00 +CPU= 36.243
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001020
15:29:09 STEP 1021 H=1021:00 +CPU= 20.647
15:29:29 STEP 1022 H=1022:00 +CPU= 20.597
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:31:14 STEP 1023 H=1023:00 +CPU= 33.210
15:31:34 STEP 1024 H=1024:00 +CPU= 20.529
15:31:54 STEP 1025 H=1025:00 +CPU= 19.903
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:33:39 STEP 1026 H=1026:00 +CPU= 33.032
15:33:58 STEP 1027 H=1027:00 +CPU= 19.716
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:35:41 STEP 1028 H=1028:00 +CPU= 21.300
15:36:14 STEP 1029 H=1029:00 +CPU= 32.773
15:36:34 STEP 1030 H=1030:00 +CPU= 19.881
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:38:06 STEP 1031 H=1031:00 +CPU= 21.227
15:38:44 STEP 1032 H=1032:00 +CPU= 37.550
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351_42.zip
Uploading the intermediate file: upload_file_42.zip
15:39:03 STEP 1033 H=1033:00 +CPU= 19.183
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:40:28 STEP 1034 H=1034:00 +CPU= 19.434
15:41:00 STEP 1035 H=1035:00 +CPU= 31.871
15:41:20 STEP 1036 H=1036:00 +CPU= 20.557
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:43:02 STEP 1037 H=1037:00 +CPU= 20.137
15:43:35 STEP 1038 H=1038:00 +CPU= 32.853
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:44:56 STEP 1039 H=1039:00 +CPU= 21.051
15:45:17 STEP 1040 H=1040:00 +CPU= 20.946
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:47:00 STEP 1041 H=1041:00 +CPU= 32.108
15:47:22 STEP 1042 H=1042:00 +CPU= 22.090
15:47:42 STEP 1043 H=1043:00 +CPU= 19.423
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:49:29 STEP 1044 H=1044:00 +CPU= 36.180
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001044
15:49:50 STEP 1045 H=1045:00 +CPU= 21.014
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:51:13 STEP 1046 H=1046:00 +CPU= 21.022
15:51:46 STEP 1047 H=1047:00 +CPU= 33.074
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:53:19 STEP 1048 H=1048:00 +CPU= 21.953
15:53:39 STEP 1049 H=1049:00 +CPU= 20.075
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:55:24 STEP 1050 H=1050:00 +CPU= 32.864
15:55:44 STEP 1051 H=1051:00 +CPU= 20.279
15:56:05 STEP 1052 H=1052:00 +CPU= 20.174
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:57:30 STEP 1053 H=1053:00 +CPU= 33.097
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:59:13 STEP 1054 H=1054:00 +CPU= 21.407
15:59:34 STEP 1055 H=1055:00 +CPU= 21.532
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:01:24 STEP 1056 H=1056:00 +CPU= 38.442
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351_43.zip
Uploading the intermediate file: upload_file_43.zip
16:02:56 STEP 1057 H=1057:00 +CPU= 20.752
16:03:16 STEP 1058 H=1058:00 +CPU= 20.467
16:03:49 STEP 1059 H=1059:00 +CPU= 32.250
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:05:21 STEP 1060 H=1060:00 +CPU= 21.190
16:05:42 STEP 1061 H=1061:00 +CPU= 20.483
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:07:26 STEP 1062 H=1062:00 +CPU= 33.302
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:08:48 STEP 1063 H=1063:00 +CPU= 20.554
16:09:08 STEP 1064 H=1064:00 +CPU= 20.528
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:10:52 STEP 1065 H=1065:00 +CPU= 32.792
16:11:14 STEP 1066 H=1066:00 +CPU= 21.880
16:11:34 STEP 1067 H=1067:00 +CPU= 19.607
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:13:22 STEP 1068 H=1068:00 +CPU= 36.827
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001068
16:13:42 STEP 1069 H=1069:00 +CPU= 20.314
16:14:03 STEP 1070 H=1070:00 +CPU= 20.405
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:15:57 STEP 1071 H=1071:00 +CPU= 32.852
16:16:17 STEP 1072 H=1072:00 +CPU= 20.360
16:16:37 STEP 1073 H=1073:00 +CPU= 19.536
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:18:10 STEP 1074 H=1074:00 +CPU= 33.218
16:18:30 STEP 1075 H=1075:00 +CPU= 19.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:20:03 STEP 1076 H=1076:00 +CPU= 20.209
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0626
(argv4) batchid: 972
(argv5) wuid: 12189270
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d8f54b1e4a7f06db78a22b9834c4bfff to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0626_2003050100_123_972_12189270.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0626_2003050100_123_972_12189270.zip
ic_ancil_file: ic_ancil_12189270
ifsdata_file: ifsdata_12189270
climate_data_file: clim_data_12189270
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_87ccf1a7e3970821feeeddf963e21501 to: /var/lib/boinc-client/slots/3/ic_ancil_12189270.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12189270.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_54f804466577be82f0c91bae5dd17ae8 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12189270.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12189270.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12189270.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12189270.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12189270.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12189270.xml
-- Model is restarting --
Adjusting last_iter, 1077, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12189270.xml
last_cpu_time: 28884
upload_file_number: 44
last_iter: 1055
last_upload: 3801600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351
The child process has been launched with process id: 973953
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:973953:973953] [20230104:162156:1672816916:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
16:22:38 STEP 1056 H=1056:00 +CPU= 36.821
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:24:18 STEP 1057 H=1057:00 +CPU= 20.344
16:24:39 STEP 1058 H=1058:00 +CPU= 20.812
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:26:23 STEP 1059 H=1059:00 +CPU= 33.236
16:26:45 STEP 1060 H=1060:00 +CPU= 22.201
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:28:15 STEP 1061 H=1061:00 +CPU= 20.042
16:28:48 STEP 1062 H=1062:00 +CPU= 33.087
16:29:09 STEP 1063 H=1063:00 +CPU= 20.231
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:30:52 STEP 1064 H=1064:00 +CPU= 21.458
16:31:26 STEP 1065 H=1065:00 +CPU= 32.931
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:33:08 STEP 1066 H=1066:00 +CPU= 21.232
16:33:28 STEP 1067 H=1067:00 +CPU= 20.399
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:35:26 STEP 1068 H=1068:00 +CPU= 36.243
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001068
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:36:49 STEP 1069 H=1069:00 +CPU= 20.942
16:37:10 STEP 1070 H=1070:00 +CPU= 20.745
16:37:41 STEP 1071 H=1071:00 +CPU= 31.225
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:39:23 STEP 1072 H=1072:00 +CPU= 20.581
16:39:43 STEP 1073 H=1073:00 +CPU= 19.956
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:41:17 STEP 1074 H=1074:00 +CPU= 33.071
16:41:37 STEP 1075 H=1075:00 +CPU= 20.206
16:41:58 STEP 1076 H=1076:00 +CPU= 20.434
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:43:52 STEP 1077 H=1077:00 +CPU= 32.520
16:44:12 STEP 1078 H=1078:00 +CPU= 20.708
16:44:33 STEP 1079 H=1079:00 +CPU= 20.185
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:46:31 STEP 1080 H=1080:00 +CPU= 37.357
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189270/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0626_2003050100_123_972_12189270_0_r1651299351_44.zip
Uploading the intermediate file: upload_file_44.zip
16:46:52 STEP 1081 H=1081:00 +CPU= 20.398
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:48:23 STEP 1082 H=1082:00 +CPU= 21.043
16:48:55 STEP 1083 H=1083:00 +CPU= 31.941
16:49:16 STEP 1084 H=1084:00 +CPU= 20.600
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:50:37 STEP 1085 H=1085:00 +CPU= 20.210
16:51:10 STEP 1086 H=1086:00 +CPU= 33.063
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:53:35 STEP 1087 H=1087:00 +CPU= 23.374
16:54:01 STEP 1088 H=1088:00 +CPU= 25.025
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:11:23 STEP 1089 H=1089:00 +CPU= 38.839
17:11:46 STEP 1090 H=1090:00 +CPU= 22.326
17:12:07 STEP 1091 H=1091:00 +CPU= 20.986
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:23:06 STEP 1092 H=1092:00 +CPU= 36.541
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001092
17:23:26 STEP 1093 H=1093:00 +CPU= 20.205
17:23:47 STEP 1094 H=1094:00 +CPU= 20.737
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:25:31 STEP 1095 H=1095:00 +CPU= 32.628
17:25:52 STEP 1096 H=1096:00 +CPU= 20.822
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:27:03 STEP 1097 H=1097:00 +CPU= 19.575
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
