climateprediction.net home page
Task 22284212

Task 22284212

Name oifs_43r3_ps_0487_2012050100_123_981_12198131_0
Workunit 12198131
Created 24 Dec 2022, 5:05:39 UTC
Sent 3 Jan 2023, 16:24:50 UTC
Report deadline 2 Feb 2023, 16:24:50 UTC
Received 4 Jan 2023, 7:43:43 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1519251
Run time 8 hours 10 min 12 sec
CPU time 8 hours 1 min 43 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.94 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.89 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,482.85 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
pending the child process
Resuming the child process
  10:55:59 STEP  809 H= 809:00 +CPU= 20.215
  10:56:32 STEP  810 H= 810:00 +CPU= 33.325
  10:56:53 STEP  811 H= 811:00 +CPU= 20.548
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:58:15 STEP  812 H= 812:00 +CPU= 21.003
  10:58:47 STEP  813 H= 813:00 +CPU= 31.965
  10:59:08 STEP  814 H= 814:00 +CPU= 20.774
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:00:20 STEP  815 H= 815:00 +CPU= 21.421
  11:00:58 STEP  816 H= 816:00 +CPU= 37.619
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404_33.zip
Uploading the intermediate file: upload_file_33.zip
  11:01:19 STEP  817 H= 817:00 +CPU= 20.309
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:02:31 STEP  818 H= 818:00 +CPU= 20.596
  11:03:03 STEP  819 H= 819:00 +CPU= 31.943
  11:03:24 STEP  820 H= 820:00 +CPU= 20.791
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:04:37 STEP  821 H= 821:00 +CPU= 20.203
  11:05:10 STEP  822 H= 822:00 +CPU= 33.045
  11:05:30 STEP  823 H= 823:00 +CPU= 20.186
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:06:44 STEP  824 H= 824:00 +CPU= 21.624
  11:07:16 STEP  825 H= 825:00 +CPU= 31.793
  11:07:37 STEP  826 H= 826:00 +CPU= 21.007
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:08:50 STEP  827 H= 827:00 +CPU= 21.073
  11:09:25 STEP  828 H= 828:00 +CPU= 35.337
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000828
  11:09:46 STEP  829 H= 829:00 +CPU= 20.576
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:10:58 STEP  830 H= 830:00 +CPU= 20.623
  11:11:30 STEP  831 H= 831:00 +CPU= 31.843
  11:11:51 STEP  832 H= 832:00 +CPU= 21.195
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:13:13 STEP  833 H= 833:00 +CPU= 20.730
  11:13:46 STEP  834 H= 834:00 +CPU= 32.425
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:14:58 STEP  835 H= 835:00 +CPU= 21.102
  11:15:19 STEP  836 H= 836:00 +CPU= 20.868
  11:15:51 STEP  837 H= 837:00 +CPU= 31.956
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:17:03 STEP  838 H= 838:00 +CPU= 21.226
  11:17:24 STEP  839 H= 839:00 +CPU= 20.994
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:18:55 STEP  840 H= 840:00 +CPU= 38.068
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404_34.zip
Uploading the intermediate file: upload_file_34.zip
  11:19:15 STEP  841 H= 841:00 +CPU= 20.594
  11:19:36 STEP  842 H= 842:00 +CPU= 20.635
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:21:00 STEP  843 H= 843:00 +CPU= 32.600
  11:21:21 STEP  844 H= 844:00 +CPU= 21.479
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:22:23 STEP  845 H= 845:00 +CPU= 20.187
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:23:36 STEP  846 H= 846:00 +CPU= 33.146
  11:23:57 STEP  847 H= 847:00 +CPU= 21.170
  11:24:18 STEP  848 H= 848:00 +CPU= 20.694
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:25:40 STEP  849 H= 849:00 +CPU= 31.974
  11:26:02 STEP  850 H= 850:00 +CPU= 21.324
  11:26:22 STEP  851 H= 851:00 +CPU= 20.112
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:27:48 STEP  852 H= 852:00 +CPU= 35.420
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000852
  11:28:08 STEP  853 H= 853:00 +CPU= 20.044
  11:28:29 STEP  854 H= 854:00 +CPU= 20.546
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:29:54 STEP  855 H= 855:00 +CPU= 32.613
  11:30:15 STEP  856 H= 856:00 +CPU= 21.510
  11:30:36 STEP  857 H= 857:00 +CPU= 20.042
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:32:21 STEP  858 H= 858:00 +CPU= 32.999
  11:32:41 STEP  859 H= 859:00 +CPU= 20.315
  11:33:02 STEP  860 H= 860:00 +CPU= 20.665
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:34:36 STEP  861 H= 861:00 +CPU= 32.758
  11:34:57 STEP  862 H= 862:00 +CPU= 21.501
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:59 STEP  863 H= 863:00 +CPU= 20.594
  11:36:38 STEP  864 H= 864:00 +CPU= 39.004
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404_35.zip
Uploading the intermediate file: upload_file_35.zip
  11:36:59 STEP  865 H= 865:00 +CPU= 20.306
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:38:28 STEP  866 H= 866:00 +CPU= 21.095
  11:39:00 STEP  867 H= 867:00 +CPU= 31.959
  11:39:21 STEP  868 H= 868:00 +CPU= 21.024
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2012050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0487
(argv4) batchid: 981
(argv5) wuid: 12198131
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ce4b2ef1e1849e877c2f5b6e98be8e80 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0487_2012050100_123_981_12198131.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0487_2012050100_123_981_12198131.zip
ic_ancil_file: ic_ancil_12198131
ifsdata_file: ifsdata_12198131
climate_data_file: clim_data_12198131
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a9563fa24c80dd5799881f5661cf1a68 to: /var/lib/boinc-client/slots/6/ic_ancil_12198131.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12198131.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4d5185cb1bd8e7691de4f2ac17008e8b to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12198131.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12198131.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12198131.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12198131.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12198131.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12198131.xml
-- Model is restarting --
Adjusting last_iter, 869, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12198131.xml
last_cpu_time: 23667
upload_file_number: 36
last_iter: 863
last_upload: 3110400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404
The child process has been launched with process id: 923548
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:923548:923548] [20230104:114228:1672800148:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:43:07 STEP  864 H= 864:00 +CPU= 36.580
  11:43:27 STEP  865 H= 865:00 +CPU= 20.249
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:44:39 STEP  866 H= 866:00 +CPU= 21.404
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:45:53 STEP  867 H= 867:00 +CPU= 33.372
  11:46:14 STEP  868 H= 868:00 +CPU= 20.536
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:47:26 STEP  869 H= 869:00 +CPU= 20.226
  11:48:00 STEP  870 H= 870:00 +CPU= 33.194
  11:48:20 STEP  871 H= 871:00 +CPU= 19.927
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:49:33 STEP  872 H= 872:00 +CPU= 21.315
  11:50:06 STEP  873 H= 873:00 +CPU= 32.561
  11:50:27 STEP  874 H= 874:00 +CPU= 20.663
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:51:30 STEP  875 H= 875:00 +CPU= 21.123
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:52:55 STEP  876 H= 876:00 +CPU= 34.628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000876
  11:53:16 STEP  877 H= 877:00 +CPU= 21.445
  11:53:37 STEP  878 H= 878:00 +CPU= 20.732
  11:54:09 STEP  879 H= 879:00 +CPU= 32.059
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:55:31 STEP  880 H= 880:00 +CPU= 21.000
  11:55:52 STEP  881 H= 881:00 +CPU= 20.301
  11:56:24 STEP  882 H= 882:00 +CPU= 32.535
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:57:46 STEP  883 H= 883:00 +CPU= 20.740
  11:58:07 STEP  884 H= 884:00 +CPU= 20.653
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:59:30 STEP  885 H= 885:00 +CPU= 32.220
  11:59:52 STEP  886 H= 886:00 +CPU= 21.589
  12:00:13 STEP  887 H= 887:00 +CPU= 20.620
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:01:52 STEP  888 H= 888:00 +CPU= 38.401
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404_36.zip
Uploading the intermediate file: upload_file_36.zip
  12:02:13 STEP  889 H= 889:00 +CPU= 20.499
  12:02:34 STEP  890 H= 890:00 +CPU= 21.150
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:03:59 STEP  891 H= 891:00 +CPU= 32.516
  12:04:20 STEP  892 H= 892:00 +CPU= 20.472
  12:04:40 STEP  893 H= 893:00 +CPU= 19.974
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:06:15 STEP  894 H= 894:00 +CPU= 33.203
  12:06:35 STEP  895 H= 895:00 +CPU= 19.987
  12:06:56 STEP  896 H= 896:00 +CPU= 21.195
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:08:21 STEP  897 H= 897:00 +CPU= 32.585
  12:08:42 STEP  898 H= 898:00 +CPU= 20.602
  12:09:02 STEP  899 H= 899:00 +CPU= 19.843
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:10:28 STEP  900 H= 900:00 +CPU= 36.054
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000900
  12:10:48 STEP  901 H= 901:00 +CPU= 19.960
  12:11:09 STEP  902 H= 902:00 +CPU= 20.540
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:12:52 STEP  903 H= 903:00 +CPU= 32.957
  12:13:13 STEP  904 H= 904:00 +CPU= 20.970
  12:13:33 STEP  905 H= 905:00 +CPU= 20.337
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:14:49 STEP  906 H= 906:00 +CPU= 33.644
  12:15:09 STEP  907 H= 907:00 +CPU= 20.224
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:16:31 STEP  908 H= 908:00 +CPU= 20.653
  12:17:04 STEP  909 H= 909:00 +CPU= 32.967
  12:17:26 STEP  910 H= 910:00 +CPU= 21.138
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:18:28 STEP  911 H= 911:00 +CPU= 21.056
  12:19:06 STEP  912 H= 912:00 +CPU= 38.047
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404_37.zip
Uploading the intermediate file: upload_file_37.zip
  12:19:26 STEP  913 H= 913:00 +CPU= 19.442
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:20:21 STEP  914 H= 914:00 +CPU= 19.887
  12:20:53 STEP  915 H= 915:00 +CPU= 32.365
  12:21:14 STEP  916 H= 916:00 +CPU= 20.892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:22:37 STEP  917 H= 917:00 +CPU= 20.370
  12:23:09 STEP  918 H= 918:00 +CPU= 32.856
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:24:22 STEP  919 H= 919:00 +CPU= 19.926
  12:24:43 STEP  920 H= 920:00 +CPU= 21.327
  12:25:16 STEP  921 H= 921:00 +CPU= 32.496
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:26:29 STEP  922 H= 922:00 +CPU= 21.277
  12:26:50 STEP  923 H= 923:00 +CPU= 20.483
  12:27:25 STEP  924 H= 924:00 +CPU= 35.515
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000924
  12:28:48 STEP  925 H= 925:00 +CPU= 20.729
  12:29:09 STEP  926 H= 926:00 +CPU= 21.057
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:30:33 STEP  927 H= 927:00 +CPU= 32.108
  12:30:54 STEP  928 H= 928:00 +CPU= 21.518
  12:31:14 STEP  929 H= 929:00 +CPU= 19.650
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:32:49 STEP  930 H= 930:00 +CPU= 33.555
  12:33:09 STEP  931 H= 931:00 +CPU= 20.067
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:34:10 STEP  932 H= 932:00 +CPU= 21.298
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:35:34 STEP  933 H= 933:00 +CPU= 32.837
  12:35:55 STEP  934 H= 934:00 +CPU= 20.717
  12:36:16 STEP  935 H= 935:00 +CPU= 20.821
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:38:05 STEP  936 H= 936:00 +CPU= 37.879
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404_38.zip
Uploading the intermediate file: upload_file_38.zip
  12:38:26 STEP  937 H= 937:00 +CPU= 20.334
  12:38:47 STEP  938 H= 938:00 +CPU= 21.009
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:40:22 STEP  939 H= 939:00 +CPU= 32.534
  12:40:42 STEP  940 H= 940:00 +CPU= 20.579
  12:41:03 STEP  941 H= 941:00 +CPU= 20.351
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:42:46 STEP  942 H= 942:00 +CPU= 32.882
  12:43:05 STEP  943 H= 943:00 +CPU= 19.720
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:44:19 STEP  944 H= 944:00 +CPU= 21.590
  12:44:52 STEP  945 H= 945:00 +CPU= 32.898
  12:45:13 STEP  946 H= 946:00 +CPU= 20.369
  12:45:33 STEP  947 H= 947:00 +CPU= 19.975
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:47:43 STEP  948 H= 948:00 +CPU= 38.543
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000948
  12:48:07 STEP  949 H= 949:00 +CPU= 23.576
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:54:04 STEP  950 H= 950:00 +CPU= 24.893
  12:54:42 STEP  951 H= 951:00 +CPU= 37.643
  12:55:07 STEP  952 H= 952:00 +CPU= 24.232
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:03:24 STEP  953 H= 953:00 +CPU= 23.969
  13:04:02 STEP  954 H= 954:00 +CPU= 37.369
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:11:36 STEP  955 H= 955:00 +CPU= 22.779
  13:11:58 STEP  956 H= 956:00 +CPU= 21.795
  13:12:31 STEP  957 H= 957:00 +CPU= 32.802
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:15:15 STEP  958 H= 958:00 +CPU= 22.345
  13:15:39 STEP  959 H= 959:00 +CPU= 23.385
  13:16:22 STEP  960 H= 960:00 +CPU= 42.308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404_39.zip
Uploading the intermediate file: upload_file_39.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:18:05 STEP  961 H= 961:00 +CPU= 23.714
  13:18:28 STEP  962 H= 962:00 +CPU= 23.237
  13:19:05 STEP  963 H= 963:00 +CPU= 35.812
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:21:09 STEP  964 H= 964:00 +CPU= 23.146
  13:21:32 STEP  965 H= 965:00 +CPU= 22.255
  13:22:09 STEP  966 H= 966:00 +CPU= 36.040
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:23:52 STEP  967 H= 967:00 +CPU= 22.534
  13:24:15 STEP  968 H= 968:00 +CPU= 23.081
  13:24:52 STEP  969 H= 969:00 +CPU= 36.268
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:27:18 STEP  970 H= 970:00 +CPU= 23.529
  13:27:41 STEP  971 H= 971:00 +CPU= 23.278
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:29:52 STEP  972 H= 972:00 +CPU= 40.035
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000972
  13:30:16 STEP  973 H= 973:00 +CPU= 23.018
  13:30:39 STEP  974 H= 974:00 +CPU= 23.174
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:32:48 STEP  975 H= 975:00 +CPU= 36.383
  13:33:10 STEP  976 H= 976:00 +CPU= 22.390
  13:33:31 STEP  977 H= 977:00 +CPU= 20.759
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2012050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0487
(argv4) batchid: 981
(argv5) wuid: 12198131
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ce4b2ef1e1849e877c2f5b6e98be8e80 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0487_2012050100_123_981_12198131.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0487_2012050100_123_981_12198131.zip
ic_ancil_file: ic_ancil_12198131
ifsdata_file: ifsdata_12198131
climate_data_file: clim_data_12198131
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a9563fa24c80dd5799881f5661cf1a68 to: /var/lib/boinc-client/slots/6/ic_ancil_12198131.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12198131.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4d5185cb1bd8e7691de4f2ac17008e8b to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12198131.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12198131.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12198131.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12198131.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12198131.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12198131.xml
-- Model is restarting --
Adjusting last_iter, 978, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12198131.xml
last_cpu_time: 26628
upload_file_number: 40
last_iter: 959
last_upload: 3456000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404
The child process has been launched with process id: 946310
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:946310:946310] [20230104:140422:1672808662:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:05:10 STEP  960 H= 960:00 +CPU= 43.628
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:18:16 STEP  961 H= 961:00 +CPU= 24.385
  14:18:42 STEP  962 H= 962:00 +CPU= 24.744
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:29:13 STEP  963 H= 963:00 +CPU= 38.315
  14:29:40 STEP  964 H= 964:00 +CPU= 25.824
  14:30:05 STEP  965 H= 965:00 +CPU= 24.139
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:41:29 STEP  966 H= 966:00 +CPU= 39.479
  14:41:53 STEP  967 H= 967:00 +CPU= 24.078
  14:42:19 STEP  968 H= 968:00 +CPU= 24.739
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:09:33 STEP  969 H= 969:00 +CPU= 38.763
  15:09:59 STEP  970 H= 970:00 +CPU= 25.847
  15:10:24 STEP  971 H= 971:00 +CPU= 24.083
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:26:45 STEP  972 H= 972:00 +CPU= 38.466
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000972
  15:27:05 STEP  973 H= 973:00 +CPU= 20.394
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:28:37 STEP  974 H= 974:00 +CPU= 20.899
  15:29:11 STEP  975 H= 975:00 +CPU= 33.655
  15:29:32 STEP  976 H= 976:00 +CPU= 20.795
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:31:04 STEP  977 H= 977:00 +CPU= 20.827
  15:31:38 STEP  978 H= 978:00 +CPU= 33.244
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:33:09 STEP  979 H= 979:00 +CPU= 20.265
  15:33:31 STEP  980 H= 980:00 +CPU= 21.449
  15:34:03 STEP  981 H= 981:00 +CPU= 31.921
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:35:45 STEP  982 H= 982:00 +CPU= 21.217
  15:36:05 STEP  983 H= 983:00 +CPU= 20.313
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:37:54 STEP  984 H= 984:00 +CPU= 37.032
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404_40.zip
Uploading the intermediate file: upload_file_40.zip
  15:38:15 STEP  985 H= 985:00 +CPU= 21.455
  15:38:35 STEP  986 H= 986:00 +CPU= 20.196
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:40:29 STEP  987 H= 987:00 +CPU= 32.033
  15:40:49 STEP  988 H= 988:00 +CPU= 20.573
  15:41:10 STEP  989 H= 989:00 +CPU= 20.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:43:05 STEP  990 H= 990:00 +CPU= 33.552
  15:43:25 STEP  991 H= 991:00 +CPU= 20.136
  15:43:46 STEP  992 H= 992:00 +CPU= 21.347
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:45:20 STEP  993 H= 993:00 +CPU= 32.451
  15:45:41 STEP  994 H= 994:00 +CPU= 20.622
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:47:13 STEP  995 H= 995:00 +CPU= 21.248
  15:47:49 STEP  996 H= 996:00 +CPU= 35.479
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000996
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:49:20 STEP  997 H= 997:00 +CPU= 19.790
  15:49:42 STEP  998 H= 998:00 +CPU= 21.960
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:51:15 STEP  999 H= 999:00 +CPU= 32.099
  15:51:37 STEP 1000 H=1000:00 +CPU= 21.719
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:53:09 STEP 1001 H=1001:00 +CPU= 21.451
  15:53:43 STEP 1002 H=1002:00 +CPU= 33.565
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:55:15 STEP 1003 H=1003:00 +CPU= 19.478
  15:55:37 STEP 1004 H=1004:00 +CPU= 21.829
  15:56:08 STEP 1005 H=1005:00 +CPU= 31.472
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:57:21 STEP 1006 H=1006:00 +CPU= 20.725
  15:57:43 STEP 1007 H=1007:00 +CPU= 21.459
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:59:43 STEP 1008 H=1008:00 +CPU= 37.983
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404_41.zip
Uploading the intermediate file: upload_file_41.zip
  16:00:02 STEP 1009 H=1009:00 +CPU= 18.679
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:01:13 STEP 1010 H=1010:00 +CPU= 20.555
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:02:58 STEP 1011 H=1011:00 +CPU= 33.275
  16:03:18 STEP 1012 H=1012:00 +CPU= 20.456
  16:03:39 STEP 1013 H=1013:00 +CPU= 20.077
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:05:23 STEP 1014 H=1014:00 +CPU= 32.906
  16:05:43 STEP 1015 H=1015:00 +CPU= 20.305
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2012050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0487
(argv4) batchid: 981
(argv5) wuid: 12198131
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ce4b2ef1e1849e877c2f5b6e98be8e80 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0487_2012050100_123_981_12198131.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0487_2012050100_123_981_12198131.zip
ic_ancil_file: ic_ancil_12198131
ifsdata_file: ifsdata_12198131
climate_data_file: clim_data_12198131
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a9563fa24c80dd5799881f5661cf1a68 to: /var/lib/boinc-client/slots/6/ic_ancil_12198131.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12198131.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4d5185cb1bd8e7691de4f2ac17008e8b to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12198131.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12198131.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12198131.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12198131.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12198131.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12198131.xml
-- Model is restarting --
Adjusting last_iter, 1016, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12198131.xml
last_cpu_time: 28105
upload_file_number: 42
last_iter: 1007
last_upload: 3628800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404
The child process has been launched with process id: 971969
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:971969:971969] [20230104:160834:1672816114:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:09:16 STEP 1008 H=1008:00 +CPU= 36.054
  16:09:35 STEP 1009 H=1009:00 +CPU= 19.768
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:11:08 STEP 1010 H=1010:00 +CPU= 21.916
  16:11:40 STEP 1011 H=1011:00 +CPU= 31.987
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:13:11 STEP 1012 H=1012:00 +CPU= 20.938
  16:13:31 STEP 1013 H=1013:00 +CPU= 20.590
  16:14:04 STEP 1014 H=1014:00 +CPU= 32.464
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:15:46 STEP 1015 H=1015:00 +CPU= 20.817
  16:16:07 STEP 1016 H=1016:00 +CPU= 20.874
  16:16:38 STEP 1017 H=1017:00 +CPU= 31.406
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:18:01 STEP 1018 H=1018:00 +CPU= 21.383
  16:18:21 STEP 1019 H=1019:00 +CPU= 20.212
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:20:09 STEP 1020 H=1020:00 +CPU= 35.793
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001020
  16:20:31 STEP 1021 H=1021:00 +CPU= 21.550
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:21:54 STEP 1022 H=1022:00 +CPU= 20.960
  16:22:27 STEP 1023 H=1023:00 +CPU= 33.209
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:24:10 STEP 1024 H=1024:00 +CPU= 21.442
  16:24:31 STEP 1025 H=1025:00 +CPU= 21.168
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:26:15 STEP 1026 H=1026:00 +CPU= 32.261
  16:26:37 STEP 1027 H=1027:00 +CPU= 21.845
  16:26:58 STEP 1028 H=1028:00 +CPU= 21.046
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:28:42 STEP 1029 H=1029:00 +CPU= 32.570
  16:29:02 STEP 1030 H=1030:00 +CPU= 20.703
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:30:44 STEP 1031 H=1031:00 +CPU= 21.005
  16:31:23 STEP 1032 H=1032:00 +CPU= 38.135
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198131/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0487_2012050100_123_981_12198131_0_r1920515404_42.zip
Uploading the intermediate file: upload_file_42.zip
  16:31:42 STEP 1033 H=1033:00 +CPU= 19.389
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:10 STEP 1034 H=1034:00 +CPU= 21.228
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:03 STEP 1035 H=1035:00 +CPU= 32.069
  16:35:25 STEP 1036 H=1036:00 +CPU= 21.657
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:48 STEP 1037 H=1037:00 +CPU= 20.593
  16:37:20 STEP 1038 H=1038:00 +CPU= 32.354
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net