climateprediction.net home page
Task 22281612

Task 22281612

Name oifs_43r3_ps_0026_2010050100_123_979_12195670_0
Workunit 12195670
Created 23 Dec 2022, 22:35:23 UTC
Sent 10 Jan 2023, 19:44:12 UTC
Report deadline 9 Feb 2023, 19:44:12 UTC
Received 13 Jan 2023, 17:00:42 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1519251
Run time 19 hours 25 min 37 sec
CPU time 19 hours 19 min 35 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 4.94 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.15 MB
Peak swap size 4,978.08 MB
Peak disk usage 3,707.94 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
request received from the BOINC client, suspending the child process
Resuming the child process
  06:24:05 STEP 2775 H=2775:00 +CPU= 33.560
  06:24:27 STEP 2776 H=2776:00 +CPU= 20.978
  06:24:47 STEP 2777 H=2777:00 +CPU= 20.433
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:26:52 STEP 2778 H=2778:00 +CPU= 33.566
  06:27:13 STEP 2779 H=2779:00 +CPU= 20.683
  06:27:35 STEP 2780 H=2780:00 +CPU= 21.599
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:29:38 STEP 2781 H=2781:00 +CPU= 32.607
  06:30:00 STEP 2782 H=2782:00 +CPU= 21.092
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:31:33 STEP 2783 H=2783:00 +CPU= 20.853
  06:32:12 STEP 2784 H=2784:00 +CPU= 39.095
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_2010050100_123_979_12195670_0_r1362559490_115.zip
Uploading the intermediate file: upload_file_115.zip
  06:32:31 STEP 2785 H=2785:00 +CPU= 19.208
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:34:09 STEP 2786 H=2786:00 +CPU= 21.503
  06:34:42 STEP 2787 H=2787:00 +CPU= 33.183
  06:35:04 STEP 2788 H=2788:00 +CPU= 21.522
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:37:16 STEP 2789 H=2789:00 +CPU= 20.683
  06:37:50 STEP 2790 H=2790:00 +CPU= 33.633
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:39:13 STEP 2791 H=2791:00 +CPU= 20.684
  06:39:33 STEP 2792 H=2792:00 +CPU= 20.512
  06:40:06 STEP 2793 H=2793:00 +CPU= 32.265
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:41:47 STEP 2794 H=2794:00 +CPU= 20.458
  06:42:08 STEP 2795 H=2795:00 +CPU= 20.176
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:43:56 STEP 2796 H=2796:00 +CPU= 36.634
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002796
  06:44:16 STEP 2797 H=2797:00 +CPU= 20.600
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:45:48 STEP 2798 H=2798:00 +CPU= 20.492
  06:46:22 STEP 2799 H=2799:00 +CPU= 33.802
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:48:35 STEP 2800 H=2800:00 +CPU= 21.258
  06:48:57 STEP 2801 H=2801:00 +CPU= 21.970
  06:49:31 STEP 2802 H=2802:00 +CPU= 33.848
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:51:23 STEP 2803 H=2803:00 +CPU= 20.331
  06:51:45 STEP 2804 H=2804:00 +CPU= 21.075
  06:52:18 STEP 2805 H=2805:00 +CPU= 32.783
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:53:50 STEP 2806 H=2806:00 +CPU= 21.210
  06:54:11 STEP 2807 H=2807:00 +CPU= 20.532
  06:54:49 STEP 2808 H=2808:00 +CPU= 37.973
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_2010050100_123_979_12195670_0_r1362559490_116.zip
Uploading the intermediate file: upload_file_116.zip
  06:55:08 STEP 2809 H=2809:00 +CPU= 19.037
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:56:50 STEP 2810 H=2810:00 +CPU= 21.038
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:58:25 STEP 2811 H=2811:00 +CPU= 33.182
  06:58:46 STEP 2812 H=2812:00 +CPU= 21.569
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:00:18 STEP 2813 H=2813:00 +CPU= 20.188
  07:00:51 STEP 2814 H=2814:00 +CPU= 32.953
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:01:53 STEP 2815 H=2815:00 +CPU= 20.837
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:03:26 STEP 2816 H=2816:00 +CPU= 21.300
  07:03:58 STEP 2817 H=2817:00 +CPU= 31.830
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:05:30 STEP 2818 H=2818:00 +CPU= 21.466
  07:05:50 STEP 2819 H=2819:00 +CPU= 20.502
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:07:38 STEP 2820 H=2820:00 +CPU= 35.730
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002820
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:09:00 STEP 2821 H=2821:00 +CPU= 20.403
  07:09:21 STEP 2822 H=2822:00 +CPU= 21.397
  07:09:54 STEP 2823 H=2823:00 +CPU= 32.241
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:11:36 STEP 2824 H=2824:00 +CPU= 21.657
  07:11:57 STEP 2825 H=2825:00 +CPU= 20.017
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:13:22 STEP 2826 H=2826:00 +CPU= 33.492
  07:13:43 STEP 2827 H=2827:00 +CPU= 20.596
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:15:14 STEP 2828 H=2828:00 +CPU= 20.981
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:16:58 STEP 2829 H=2829:00 +CPU= 32.776
  07:17:19 STEP 2830 H=2830:00 +CPU= 21.439
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:18:42 STEP 2831 H=2831:00 +CPU= 21.109
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:20:12 STEP 2832 H=2832:00 +CPU= 38.675
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_2010050100_123_979_12195670_0_r1362559490_117.zip
Uploading the intermediate file: upload_file_117.zip
  07:20:32 STEP 2833 H=2833:00 +CPU= 20.462
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:21:39 STEP 2834 H=2834:00 +CPU= 20.394
  07:22:11 STEP 2835 H=2835:00 +CPU= 32.273
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:23:43 STEP 2836 H=2836:00 +CPU= 20.488
  07:24:04 STEP 2837 H=2837:00 +CPU= 21.499
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:25:38 STEP 2838 H=2838:00 +CPU= 33.223
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:26:58 STEP 2839 H=2839:00 +CPU= 20.681
  07:27:19 STEP 2840 H=2840:00 +CPU= 20.987
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:29:04 STEP 2841 H=2841:00 +CPU= 32.305
  07:29:26 STEP 2842 H=2842:00 +CPU= 21.752
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:30:47 STEP 2843 H=2843:00 +CPU= 20.514
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:32:25 STEP 2844 H=2844:00 +CPU= 36.504
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002844
  07:32:46 STEP 2845 H=2845:00 +CPU= 20.650
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:34:18 STEP 2846 H=2846:00 +CPU= 20.911
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:35:42 STEP 2847 H=2847:00 +CPU= 33.330
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:37:15 STEP 2848 H=2848:00 +CPU= 21.089
  07:37:35 STEP 2849 H=2849:00 +CPU= 20.361
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:40:10 STEP 2850 H=2850:00 +CPU= 33.163
  07:40:31 STEP 2851 H=2851:00 +CPU= 20.693
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:42:14 STEP 2852 H=2852:00 +CPU= 21.374
  07:42:46 STEP 2853 H=2853:00 +CPU= 32.165
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:44:49 STEP 2854 H=2854:00 +CPU= 21.489
  07:45:09 STEP 2855 H=2855:00 +CPU= 20.872
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:47:00 STEP 2856 H=2856:00 +CPU= 38.673
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_2010050100_123_979_12195670_0_r1362559490_118.zip
Uploading the intermediate file: upload_file_118.zip
  07:47:19 STEP 2857 H=2857:00 +CPU= 18.506
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:48:26 STEP 2858 H=2858:00 +CPU= 20.972
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:50:00 STEP 2859 H=2859:00 +CPU= 33.193
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:51:33 STEP 2860 H=2860:00 +CPU= 21.280
  07:51:54 STEP 2861 H=2861:00 +CPU= 21.587
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:53:19 STEP 2862 H=2862:00 +CPU= 34.604
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:54:41 STEP 2863 H=2863:00 +CPU= 20.775
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:56:05 STEP 2864 H=2864:00 +CPU= 21.815
  07:56:38 STEP 2865 H=2865:00 +CPU= 32.517
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:58:10 STEP 2866 H=2866:00 +CPU= 20.858
  07:58:30 STEP 2867 H=2867:00 +CPU= 20.178
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:00:08 STEP 2868 H=2868:00 +CPU= 36.492
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002868
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0026
(argv4) batchid: 979
(argv5) wuid: 12195670
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_75402b9259fe5d2bdf564213787f7884 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0026_2010050100_123_979_12195670.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0026_2010050100_123_979_12195670.zip
ic_ancil_file: ic_ancil_12195670
ifsdata_file: ifsdata_12195670
climate_data_file: clim_data_12195670
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a9f0cb7682b4e38df14999ab41d0e971 to: /var/lib/boinc-client/slots/0/ic_ancil_12195670.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12195670.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3748859929ec4965f83945320a644dfd to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12195670.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12195670.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12195670.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12195670.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12195670.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12195670.xml
-- Model is restarting --
Adjusting last_iter, 2869, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12195670.xml
last_cpu_time: 66367
upload_file_number: 119
last_iter: 2855
last_upload: 10281600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0026_2010050100_123_979_12195670_0_r1362559490
The child process has been launched with process id: 1382834
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1382834:1382834] [20230112:080345:1673478225:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:05:29 STEP 2856 H=2856:00 +CPU= 38.258
  08:05:50 STEP 2857 H=2857:00 +CPU= 20.527
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:07:12 STEP 2858 H=2858:00 +CPU= 21.562
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:08:47 STEP 2859 H=2859:00 +CPU= 32.411
  08:09:08 STEP 2860 H=2860:00 +CPU= 21.093
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:10:50 STEP 2861 H=2861:00 +CPU= 20.926
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:12:34 STEP 2862 H=2862:00 +CPU= 33.676
  08:12:56 STEP 2863 H=2863:00 +CPU= 21.607
  08:13:16 STEP 2864 H=2864:00 +CPU= 20.763
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:15:21 STEP 2865 H=2865:00 +CPU= 32.688
  08:15:42 STEP 2866 H=2866:00 +CPU= 20.994
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:17:43 STEP 2867 H=2867:00 +CPU= 20.009
  08:18:19 STEP 2868 H=2868:00 +CPU= 34.964
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002868
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:19:49 STEP 2869 H=2869:00 +CPU= 19.993
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:20:52 STEP 2870 H=2870:00 +CPU= 22.054
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:22:37 STEP 2871 H=2871:00 +CPU= 32.559
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:24:07 STEP 2872 H=2872:00 +CPU= 20.809
  08:24:29 STEP 2873 H=2873:00 +CPU= 21.125
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:26:13 STEP 2874 H=2874:00 +CPU= 33.055
  08:26:34 STEP 2875 H=2875:00 +CPU= 20.908
  08:26:55 STEP 2876 H=2876:00 +CPU= 20.563
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:28:58 STEP 2877 H=2877:00 +CPU= 32.240
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:30:30 STEP 2878 H=2878:00 +CPU= 20.992
  08:30:51 STEP 2879 H=2879:00 +CPU= 21.173
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:32:51 STEP 2880 H=2880:00 +CPU= 37.988
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_2010050100_123_979_12195670_0_r1362559490_119.zip
Uploading the intermediate file: upload_file_119.zip
  08:33:10 STEP 2881 H=2881:00 +CPU= 19.329
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:34:53 STEP 2882 H=2882:00 +CPU= 20.539
  08:35:26 STEP 2883 H=2883:00 +CPU= 33.164
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:37:08 STEP 2884 H=2884:00 +CPU= 21.405
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:38:41 STEP 2885 H=2885:00 +CPU= 21.139
  08:39:14 STEP 2886 H=2886:00 +CPU= 33.186
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:40:46 STEP 2887 H=2887:00 +CPU= 20.936
  08:41:08 STEP 2888 H=2888:00 +CPU= 21.718
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:42:52 STEP 2889 H=2889:00 +CPU= 33.325
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:44:16 STEP 2890 H=2890:00 +CPU= 21.925
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:46:17 STEP 2891 H=2891:00 +CPU= 20.730
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:47:46 STEP 2892 H=2892:00 +CPU= 36.375
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002892
  08:48:07 STEP 2893 H=2893:00 +CPU= 21.180
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:49:29 STEP 2894 H=2894:00 +CPU= 21.207
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:51:23 STEP 2895 H=2895:00 +CPU= 32.850
  08:51:45 STEP 2896 H=2896:00 +CPU= 21.905
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:53:57 STEP 2897 H=2897:00 +CPU= 21.679
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:56:13 STEP 2898 H=2898:00 +CPU= 34.570
  08:56:35 STEP 2899 H=2899:00 +CPU= 21.798
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:58:28 STEP 2900 H=2900:00 +CPU= 21.434
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:00:23 STEP 2901 H=2901:00 +CPU= 33.277
  09:00:45 STEP 2902 H=2902:00 +CPU= 21.661
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:02:27 STEP 2903 H=2903:00 +CPU= 21.403
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:04:37 STEP 2904 H=2904:00 +CPU= 38.278
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_2010050100_123_979_12195670_0_r1362559490_120.zip
Uploading the intermediate file: upload_file_120.zip
  09:04:58 STEP 2905 H=2905:00 +CPU= 21.091
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:06:34 STEP 2906 H=2906:00 +CPU= 20.517
  09:07:07 STEP 2907 H=2907:00 +CPU= 32.705
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:09:11 STEP 2908 H=2908:00 +CPU= 22.670
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:11:14 STEP 2909 H=2909:00 +CPU= 21.742
  09:11:48 STEP 2910 H=2910:00 +CPU= 33.939
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:13:31 STEP 2911 H=2911:00 +CPU= 21.860
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:28:56 STEP 2912 H=2912:00 +CPU= 21.759
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:31:01 STEP 2913 H=2913:00 +CPU= 33.373
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:33:15 STEP 2914 H=2914:00 +CPU= 22.032
  09:33:36 STEP 2915 H=2915:00 +CPU= 21.046
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:35:53 STEP 2916 H=2916:00 +CPU= 35.991
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002916
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:37:46 STEP 2917 H=2917:00 +CPU= 20.926
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:39:19 STEP 2918 H=2918:00 +CPU= 21.339
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:41:33 STEP 2919 H=2919:00 +CPU= 33.035
  09:41:55 STEP 2920 H=2920:00 +CPU= 21.928
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:43:57 STEP 2921 H=2921:00 +CPU= 21.246
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:46:45 STEP 2922 H=2922:00 +CPU= 34.834
  09:47:06 STEP 2923 H=2923:00 +CPU= 21.460
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:49:29 STEP 2924 H=2924:00 +CPU= 21.642
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:51:54 STEP 2925 H=2925:00 +CPU= 33.497
  09:52:17 STEP 2926 H=2926:00 +CPU= 22.296
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:54:49 STEP 2927 H=2927:00 +CPU= 21.083
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:56:40 STEP 2928 H=2928:00 +CPU= 39.232
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_2010050100_123_979_12195670_0_r1362559490_121.zip
Uploading the intermediate file: upload_file_121.zip
  09:57:01 STEP 2929 H=2929:00 +CPU= 20.715
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:58:54 STEP 2930 H=2930:00 +CPU= 21.355
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:01:19 STEP 2931 H=2931:00 +CPU= 33.059
  10:01:40 STEP 2932 H=2932:00 +CPU= 21.533
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0026
(argv4) batchid: 979
(argv5) wuid: 12195670
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_75402b9259fe5d2bdf564213787f7884 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0026_2010050100_123_979_12195670.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0026_2010050100_123_979_12195670.zip
ic_ancil_file: ic_ancil_12195670
ifsdata_file: ifsdata_12195670
climate_data_file: clim_data_12195670
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a9f0cb7682b4e38df14999ab41d0e971 to: /var/lib/boinc-client/slots/0/ic_ancil_12195670.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12195670.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3748859929ec4965f83945320a644dfd to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12195670.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12195670.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12195670.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12195670.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12195670.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12195670.xml
-- Model is restarting --
Adjusting last_iter, 2933, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12195670.xml
last_cpu_time: 68349
upload_file_number: 122
last_iter: 2927
last_upload: 10540800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0026_2010050100_123_979_12195670_0_r1362559490
The child process has been launched with process id: 1446187
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1446187:1446187] [20230112:113827:1673491107:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:41:29 STEP 2928 H=2928:00 +CPU= 37.433
  11:41:51 STEP 2929 H=2929:00 +CPU= 21.952
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:44:04 STEP 2930 H=2930:00 +CPU= 21.668
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:46:28 STEP 2931 H=2931:00 +CPU= 32.507
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:48:41 STEP 2932 H=2932:00 +CPU= 21.515
  11:49:01 STEP 2933 H=2933:00 +CPU= 20.742
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:51:46 STEP 2934 H=2934:00 +CPU= 33.526
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:53:39 STEP 2935 H=2935:00 +CPU= 21.254
  11:54:01 STEP 2936 H=2936:00 +CPU= 21.698
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:57:06 STEP 2937 H=2937:00 +CPU= 33.891
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:59:19 STEP 2938 H=2938:00 +CPU= 21.488
  11:59:40 STEP 2939 H=2939:00 +CPU= 21.267
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:02:38 STEP 2940 H=2940:00 +CPU= 36.892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002940
  12:05:20 STEP 2941 H=2941:00 +CPU= 20.932
  12:05:42 STEP 2942 H=2942:00 +CPU= 21.715
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:09:26 STEP 2943 H=2943:00 +CPU= 33.286
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:11:39 STEP 2944 H=2944:00 +CPU= 21.844
  12:12:00 STEP 2945 H=2945:00 +CPU= 20.958
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:22:49 STEP 2946 H=2946:00 +CPU= 34.330
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:52:35 STEP 2947 H=2947:00 +CPU= 21.371
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:56:48 STEP 2948 H=2948:00 +CPU= 21.720
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:05:04 STEP 2949 H=2949:00 +CPU= 33.169
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0026
(argv4) batchid: 979
(argv5) wuid: 12195670
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_75402b9259fe5d2bdf564213787f7884 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0026_2010050100_123_979_12195670.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0026_2010050100_123_979_12195670.zip
ic_ancil_file: ic_ancil_12195670
ifsdata_file: ifsdata_12195670
climate_data_file: clim_data_12195670
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a9f0cb7682b4e38df14999ab41d0e971 to: /var/lib/boinc-client/slots/0/ic_ancil_12195670.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12195670.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3748859929ec4965f83945320a644dfd to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12195670.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12195670.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12195670.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12195670.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12195670.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12195670.xml
-- Model is restarting --
Adjusting last_iter, 2949, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12195670.xml
last_cpu_time: 68923
upload_file_number: 122
last_iter: 2927
last_upload: 10540800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0026_2010050100_123_979_12195670_0_r1362559490
The child process has been launched with process id: 1524551
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1524551:1524551] [20230112:150748:1673503668:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:11:02 STEP 2928 H=2928:00 +CPU= 37.479
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:13:45 STEP 2929 H=2929:00 +CPU= 21.302
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:16:58 STEP 2930 H=2930:00 +CPU= 21.474
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:20:03 STEP 2931 H=2931:00 +CPU= 33.115
  15:20:24 STEP 2932 H=2932:00 +CPU= 20.922
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:23:06 STEP 2933 H=2933:00 +CPU= 21.108
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:28:12 STEP 2934 H=2934:00 +CPU= 33.486
  15:28:33 STEP 2935 H=2935:00 +CPU= 20.868
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:31:06 STEP 2936 H=2936:00 +CPU= 21.213
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:34:10 STEP 2937 H=2937:00 +CPU= 32.236
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:37:13 STEP 2938 H=2938:00 +CPU= 21.960
  15:37:34 STEP 2939 H=2939:00 +CPU= 20.713
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:43:12 STEP 2940 H=2940:00 +CPU= 36.766
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002940
  15:43:32 STEP 2941 H=2941:00 +CPU= 20.242
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:46:45 STEP 2942 H=2942:00 +CPU= 21.604
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:49:39 STEP 2943 H=2943:00 +CPU= 32.856
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:52:32 STEP 2944 H=2944:00 +CPU= 21.826
  15:52:53 STEP 2945 H=2945:00 +CPU= 20.551
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:57:18 STEP 2946 H=2946:00 +CPU= 33.463
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:00:30 STEP 2947 H=2947:00 +CPU= 20.534
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:58 STEP 2948 H=2948:00 +CPU= 21.409
  16:36:31 STEP 2949 H=2949:00 +CPU= 33.127
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:44:24 STEP 2950 H=2950:00 +CPU= 21.753
  16:44:45 STEP 2951 H=2951:00 +CPU= 20.966
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:56:38 STEP 2952 H=2952:00 +CPU= 38.146
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195670/ICMUAhq0f+002928
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_2010050100_123_979_12195670_0_r1362559490_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
17:00:02 (1524541): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:16:29 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 10,623,600 69,575 0.0065
09 May 2023 16:15:48 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 9,504,000 60,170 0.0063
09 May 2023 16:15:45 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 8,640,000 55,282 0.0064
09 May 2023 16:15:45 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 8,640,000 55,282 0.0064
09 May 2023 16:15:41 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 7,776,000 50,394 0.0065
09 May 2023 16:15:38 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 6,912,000 45,507 0.0066
09 May 2023 16:15:23 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 6,048,000 39,627 0.0066
09 May 2023 16:15:19 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 5,184,000 33,410 0.0064
09 May 2023 16:15:14 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 4,320,000 27,432 0.0064
09 May 2023 16:15:12 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 3,456,000 21,751 0.0063
09 May 2023 16:15:12 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 3,456,000 21,751 0.0063
09 May 2023 16:15:08 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 2,592,000 16,131 0.0062
09 May 2023 16:15:08 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 2,592,000 16,131 0.0062
09 May 2023 16:15:06 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 1,728,000 10,759 0.0062
09 May 2023 16:15:06 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 1,728,000 10,759 0.0062
09 May 2023 16:15:02 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 864,000 5,418 0.0063
09 May 2023 16:15:02 1519251 22281612 oifs_43r3_ps_0026_2010050100_123_979_12195670_0 864,000 5,418 0.0063


©2024 climateprediction.net